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Abstract 

Different aspects of the predictability problem in dynamical systems are reviewed. 
The deep relation among Lyapunov exponents, Kolmogorov- Sinai entropy, Shan- 
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non entropy and algorithmic complexity is discussed. In particular, we emphasize 
how a characterization of the unpredictability of a system gives a measure of its 
complexity. Adopting this point of view, we review some developments in the char- 
acterization of the predictability of systems showing different kind of complexity: 
from low-dimensional systems to high-dimensional ones with spatio-temporal chaos 
and to fully developed turbulence. A special attention is devoted to finite-time 
and finite-resolution effects on predictability, which can be accounted with suitable 
generalization of the standard indicators. The problems involved in systems with 
intrinsic randomness is discussed, with emphasis on the important problems of dis- 
tinguishing chaos from noise and of modeling the system. The characterization of 
irregular behavior in systems with discrete phase space is also considered. 

PACS numbers: 
Key words: 
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All the simple systems are simple in the same way, each complex system has 
its own complexity 
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1 Introduction 

The ability to predict the future state of a system, given the present one, stands 
at the foundations of scientific knowledge with relevant implications from a 
conceptual and applicative point of view. The knowledge of the evolution law 
of the system may induce one to conclude that this aim has been attained. This 
is the classical deterministic point of view as clearly stated by Laplace [134]: 
once the evolution laws of the system are known, the state at a certain time to 
completely determines the subsequent states for every time t > toQ. However 
it is well established now that this cannot be accomplished in practice. 

One limitation occurs in systems with many degrees of freedom, namely the 
impossibility to manage the huge amount of data required for a detailed de- 
scription of a single state of a macroscopic body. This aspect, which is not 
discussed in this review, has led to the development of statistical mechanics. 

Another source of difficulty, which arises even in low dimensional systems, 
is related to the unavoidable uncertainty in the initial condition. As clearly 
stated by Poincare, this implies that one can make long-time predictions only 
if the evolution law does not amplify the initial uncertainty too rapidly. This 
aspect had a relevant role in the development of the theory of dynamical chaos. 

Therefore, from the point of view of predictability, we need to know how an 
error in the initial state of the system grows in time. In deterministic chaotic 
systems, i.e., with sensitive dependence on initial condition, one has an expo- 
nential growth of errors and, consequently, a severe limitation on the ability 
to predict the future states. In addition, since the details of the evolution laws 
are not completely known (or, at least, cannot be specified with an arbitrary 
accuracy) or some degrees of freedom cannot be resolved, one has another un- 
avoidable source of unpredictability. This is also true for systems with discrete 
states. 

A branch of the theory of dynamical systems has been developed with the aim 
of formalizing and quantitatively characterizing the sensitivity to initial condi- 
tions. The Lyapunov exponent and the Kolmogorov- Sinai entropy are the two 
indicators for measuring the rate of error growth and information produced 
by the dynamical system. A complementary approach has been developed in 
the context of information theory, data compression and algorithmic com- 
plexity theory. Nowadays it is rather clear that the latter point of view is 
closely related to the dynamical systems one. If a system is chaotic then the 
predictability is limited up to a time which is related to the first Lyapunov 
exponent, and the time sequence generated from one of its chaotic trajectories 

^ In this review we shall always consider the usual setting where a system is studied 
by an external observer, so as to avoid the problem of the self-prediction [192]. 
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cannot be compressed by an arbitrary factor, i.e. is algorithmically complex. 

On the contrary, a regular trajectory can be easily compressed (e.g., for a 
periodic trajectory it is sufficient to have the sequence for a period) so it is 
"simple" . 

In this review we will discuss how these points of view are related and how 
they complete each other in giving a quantitative understanding of complexity 
arising in dynamical systems. In particular, we shall consider the extension of 
this approach, nowadays well established in the context of low dimensional 
systems and for asymptotic regimes, to high dimensional systems with atten- 
tion to situations far from asymptotic (i.e. finite time and finite observational 
resolution) . 

It is worth remarking that the approach to complexity here discussed is not 
able to cover the many aspects of what in recent years has been indicated 
under this term [13]. Indeed complexity has been considered in many different 
fields of science and its meaning has become (sometimes) vague. A challenging 
problem in this direction is the definition of indicators which are able to fulfill 
the intuitive idea of complexity, namely one looks for quantities which give a 
low complexity value both for pure random sequences and completely regular 
ones [96]. Even if very interesting this last issue is not addressed here: from 
the point of view of predictability both a chaotic system and a purely random 
one are highly complex, i.e. unpredictable. 

The review is organized as follows. Sect. 2 is devoted to the introduction of 
the basic concepts and ideas of dynamical systems, information theory and al- 
gorithmic complexity. In particular, we discuss the relations among Lyapunov 
exponents, Kolmogorov-Sinai entropy and algorithmic complexity and their 
relevance for predictability. All these quantities are properly defined only in 
specific asymptotic limits, that are: very long times and arbitrary accuracy. 
Since in realistic situations one has to deal with finite accuracy and finite time 
— as Keynes said, "in the long run we shall all be dead" — it is appealing 
to treat the predictability problem by taking into account these limitations. 
This is the subject of Sect. 3 where, relaxing the request of infinite time, we 
discuss the relevance of the finite time fluctuations of the "effective" Lyapunov 
exponent. In addition, relaxing the limit of infinitesimal perturbations, we in- 
troduce suitable tools, such as the Finite Size Lyapunov Exponent (FSLE) 
and the e-entropy, for the treatment of non arbitrary high accuracy, i.e. non 
infinitesimal perturbations. 

Sects. 4 and 5 focus on high dimensional dynamical systems which deserve 

particular attention. Indeed because of the many degrees of freedom, and 
its interest in applications (e.g. in weather forecasting), it is necessary to 
consider the detailed behavior of perturbations and not only the asymptotic 
features (i.e. long time and infinitesimal amplitudes). Sect. 5 is devoted to 
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fully developed turbulence (here introduced as an important prototype of high 
dimensional system) and its finite resolution properties in the inertial range. 

In Sect. 6 we consider the effects of uncertainties on the evolution laws and 
we discuss systems containing some randomness. In such a situation there are 
two ways to approach the predictability: by considering either two trajecto- 
ries generated with the same realization of randomness, or two trajectories 
evolving with different realizations. Both approaches are physically relevant 
in different contexts, and the results can be very different in presence of strong 
intermittency. 

For the sake of completeness in Sect. 7 we discuss dynamical systems with 
discrete states, e.g.. Cellular Automata. 

Sect. 8 is dedicated to a discussion on data analysis. In particular we discuss 
the use of e-entropy and FSLE for a pragmatic classification of signals. 

Sect. 9 reports some concluding remarks. In the Appendices we discuss some 
more technical details. 



2 Two points of view 

2.1 Dynamical systems approach 

Two standard - tightly linked - indicators are largely used to quantify the 
behavior of a dynamical system with respect to the asymptotic evolution of 
an infinitesimal uncertainty: the largest Lyapunov exponent (LE) and the 
Kolmogorov- Sinai (or metric) entropy [74]. 

2.1.1 Characteristic Lyapunov exponents 

The characteristic Lyapunov exponents are somehow an extension of the linear 
stability analysis to the case of aperiodic motions. Roughly speaking, they 
measure the typical rate of exponential divergence of nearby trajectories. In 
this sense they give information on the rate of growth of a very small error on 
the initial state of a system. 

Consider a dynamical system with an evolution law given, in the case of con- 
tinuous time, by the differential equation 

$=FW, (2.1) 
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or, in the case of discrete time, by the map 



x(t+l) = G(x(t)). (2.2) 

In both cases, for simphcity, we suppose that a vector x G H*^ uniquely spec- 
ifies one state of the system. We also assume that F and G are differentiable 
functions, that the evolution is well-defined for time intervals of arbitrary ex- 
tension, and that the motion takes place in a bounded region of the phase 
space. We intend to study the separation between two trajectories, x(t) and 
x'(t), starting from two close initial conditions, x(0) and x'(0) = x(0) -|-(5x(0), 
respectively. 

As long as the difference between the trajectories, 5x(i) = x'(t) — x(i), remains 
small (infinitesimal, strictly speaking), it can be regarded as a vector. z(t), 
in the tangent space. The time evolution of z{t) is given by the linearized 
differential equations: 

dzjjt) ^ ^ m 

dt dxj 
or, in the case of discrete time maps: 



z,{t). (2.4) 

x(t) 

Under rather general hypothesis, Oseledec [169] proved that for almost all 
initial conditions x(0), there exists an orthonormal basis {ej} in the tangent 
space such that, for large times, 

d 

z(^)=Ec^e^e^^^ (2.5) 

i=l 

where the coefficients {cj} depends on z(0). The exponents Ai > A2 > • • • > 
are called characteristic Lyapunov exponents. If the dynamical system has an 
ergodic invariant measure, the spectrum of LEs {Aj} does not depend on the 
initial condition, except for a set of measure zero with respect to the natural 
invariant measure. 

Loosely speaking, (2.5) tells us that in the phase space, where the motion 
evolves, a dimensional sphere of small radius e centered in x(0) is deformed 
with time into an ellipsoid of semi-axes ei{t) — eexp(Aji), directed along the 



cm 



(2.3) 



/ N X — > dGj 

^^(^ + i) = E 5^ 
7=1 '-'•^j 
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Gj vectors. Furthermore, for a generic small perturbation 5x(0), the distance 
between a trajectory and the perturbed one behaves as 



-(Ai-A2)t 



(2.6) 



If Ai > we have a rapid (exponential) amplification of an error on the initial 
condition. In such a case, the system is chaotic and, de facto, unpredictable 
on the long times. Indeed, if we put Sq — |(^x(0)| for the initial error, and we 
want to predict the states of the system with a certain tolerance A (not too 
large), then the prediction is possible just up to a predictability time given by 

T.-h-(T)- (2-7) 



This equation shows that Tp is basically determined by the largest Lyapunov 
exponent, since its dependence on Sq and A is very weak. Because of its pre- 
eminent role, very often one simply refers to Ai as "the Lyapunov exponent" , 
and one indicates it with A. 



Eq. (2.6) suggests how to numerically compute Ai. We introduce the response, 
after a time t, to a perturbation on x(t), defined as follows: 

|z(r)| |5x(r)| 



where, again, |5x(r)| and |5x(r + t)| are infinitesimal. The LE Ai is obtained 
by averaging the logarithm of the response over the initial conditions or along 
the trajectory: 

Xi^ Urn -{In Rr(t)), (2.9) 

t— >oo / 



where (•) denotes the time average limT^oo(l/T)/;°+'(-)c^T. The Oseledec's 
theorem implies that (l/t) lni?^(i), for large t, is a non-random quantity, i.e. 
for almost all the initial conditions its value does not depend on the spe- 
cific initial condition. Therefore, for large times, the average in (2.9) can be 
neglected. 

As the typical growth rate of a generic small segment in phase space is driven 
by the largest LE, the sum of the first n (< d) Lyapunov exponents controls 
the variations of small n-dimensional volumes in phase space. This gives us 
a way to compute the sub-leading Lyapunov exponents. After the selection 
oi n < d non parallel tangent vectors [z'^^)(0), . . . , z(")(0)], one introduces the 
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n-order response R^^t) [20] 



zi(t + r) X Z2(t + r) X • • • X z„(t + t)| 
|zi(r) X Z2(r) X • • • X z„(r)| 



(2.10) 



Analogously to the LE, it can be shown that 



X:A, = ^liml(lni?(")(t)). 



(2.11) 



Let us stress that the Lyapunov exponents give information on the typical 
behaviors along a generic trajectory, followed for infinite time and keeping the 
perturbation infinitesimally small. In this respect, they are global quantities 
characterizing fine-grained properties of a system. 

2.1.2 Kolmogorov-Sinai entropy 

The LE, A, gives a first quantitative information on how rapidly we loose the 
ability of predicting the evolution of a system. A state, initially determined 
with an error 5x(0), after a time enough larger than 1/A, may be found almost 
everywhere in the region of motion. In this respect, the Kolmogorov-Sinai (KS) 
entropy, hKSi supplies a more refined information. The error on the initial 
state is due to the maximal resolution we use for observing the system. For 
simplicity, let us assume the same resolution e for each degree of freedom. 
We build a partition of the phase space with cells of volume e*^, so that the 
state of the system aX, t — is found in a region of volume Vq — e'^ around 
x(fo). Now we consider the trajectories starting from Vq at to and sampled 
at discrete times tj — jr {j — 1, 2, 3, . . . , t): in the case of a map one can 
put r = 1. Since we are considering motions that evolve in a bounded region, 
all the trajectories visit a finite number of different cells, each one identified 
by a symbol. In this way a unique sequence of symbols {s(0), s(l), s{2), . . .} is 
associated with a given trajectory. In a chaotic system, although each evolution 
x(i) is univocally determined by x(io), a great number of different symbolic 
sequences originates by the same initial cell, because of the divergence of 
nearby trajectories. The total number of the admissible symbohc sequences, 
N{e,t), increases exponentially with a rate given by the topological entropy 



which appear with very high probability in the long time limit - those that 
can be numerically or experimentally detected and that are associated with the 



hr — lim lim - In A^(e, 

e^Ot^oo t 



(2.12) 



However, if we consider only the number of sequences 
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natural measure - we arrive at a more physical quantity called the Kolmogorov- 
Sinai or metric entropy [74]: 

Hks = lim^lim ^lniVe//(e,t) < hr- (2.13) 

hKS quantifies the long time exponential rate of growth of the number of 
the effective coarse-grained trajectories of a system. This suggests a link with 
information theory where the Shannon entropy measures the mean asymptotic 
growth of the number of the typical sequences - the ensemble of which has 
probability almost one - emitted by a source. In the following we will discuss 
in more detail the KS-entropy and its relation with the information theory. 
Here we obtain, by means of a heuristic reasoning, the relation among hxs 
and Lyapunov exponents. 

We may wonder what is the number of cells where, at a time t > to, the 
points that evolved from Vq can be found, i.e. we wish to know how big is 
the coarse-grained volume V{e,t), occupied by the states evolved from Vq, 
if the minimum volume we can observe is e'^. As stated at the end of the 
preceding subsection, we have V{t) ~ Vq exp(t J2i=i K)- However, this is true 
only in the limit e ^ 0. In this (unrealistic) limit, V{t) = Vq for a conservative 
system (where Y.i=i \ = 0) and V{t) < Vq for a dissipative system (where 
J2i=iK < 0). As a consequence of Umited resolution power, in the evolution 
of the volume Vq — e*^ the effect of the contracting directions (associated with 
the negative Lyapunov exponents) is completely lost. We can experience only 
the effect of the expanding directions, associated with the positive Lyapunov 
exponents. As a consequence, in the typical case, the coarse grained volume 
behaves as 

V{e,t) r^Voe^^^i>'>^'^\ (2.14) 

when Vo is small enough. Since Neff{e,t) oc V{€,t)/Vo, one has 

hKS^J2K. (2.15) 

Ai>0 

This argument can be made more rigorous with a proper mathematical defini- 
tion of the metric entropy. In this case one derives the Pesin relation [179,74] 

hKS < E ^i- (2-16) 

Ai>0 

Because of its relation with the Lyapunov exponents - or by the definition 
(2.13) - it is clear that also hxs is a fine-grained and global characterization 
of a dynamical system. 
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The metric entropy is an invariant characteristic quantity of a dynamical 
system [125,204], i.e. given two systems with invariant measures, their KS- 
entropies exist and they arc equal if the systems are isomorphic [31]. This 
intrinsic quantity may be properly defined by means of tools borrowed from 
the mathematical theory of communication. 

2.2 Information theory approach 

In experimental investigations of physical processes, we typically have access 
to the system only trough a measuring device which produces a time record of 
a certain observable, i.e. a sequence of data. In this regard a system, whether 
or not chaotic, generates messages and may be regarded as a source of in- 
formation. This observation opens the possibility to study dynamical systems 
from a very interesting point of view. 

Information has found a proper characterization in the framework of the the- 
ory of communication to cope with the practical problem of transmitting a 

message in the cheapest way without losing information. The characterization 
of the information contained in a sequence can be approached by two very 
different points of view. The first one, that of information theory [201], is a 
statistical approach, i.e., it does not consider the transmission of a specific 
message (sequence) but refers to the statistical properties of all the messages 
emitted by the source. Information theory approach characterizes the source 
of information, so that it gives us a powerful method to characterize chaotic 
systems. 

The second point of view considers the problem of characterizing a single 
sequence. This latter has led to the theory of algorithmic complexity and 
algorithmic information theory [53,126,207]. 

2.2.1 Shannon entropy 

At the end of forties Shannon [201] introduced rather powerful concepts and 
techniques for a systematic study of sources emitting sequences of discrete 
symbols (e.g. binary digit sequences). Originally information theory was in- 
troduced in the practical context of electric communications, nevertheless in a 
few years it became an important branch of both pure and applied probability 
theory with strong relations with other fields as computer science, cryptogra- 
phy, biology and physics [230]. 

For the sake of self-consistency we briefly recall the basic concepts and ideas 
about the Shannon entropy. Consider a source that can output m different 
symbols; denote with s{t) the symbol emitted by the source at time t and 
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with P{Cn) the probabihty that a given word Cn — (^(l), s(2), . . . , s{N)), of 
length N, is emitted: 



P{CN)=Pis{l),s{2),...,s{N)). (2.17) 

We assume that the source is stationary, so that for the sequences {<s(t)} the 
time translation invariance holds: P(s(l), . . . , s{N)) = P{s{t + 1), . . . ,s{t + 
N)). 

Now we introduce the iV-block entropies 

/7jv = - E Pi^N) InP(Cjv) , (2.18) 

{Cn} 

and the differential entropies 

Hn — Hn+i — Hn, (2.19) 

whose meaning is the average information supplied by the (A^ + l)-th symbol, 
provided the N previous ones are known. One can also say that is the 
average uncertainty about the {N + l)-th symbol, provided the N previous 
ones are given. For a stationary source the limits in the following equations 
exist, are equal and define the Shannon entropy hsh'- 

hsh = lim hN = lim — f . (2.20) 

The are non increasing quantities: Hn+i < h^; that is: the knowledge of 
a longer past history cannot increase the uncertainty on the next outcome. 
In the case of a k-th order Markov process Hn = hsh for all N > k. This is 
because a k-th order Markov process has the property that the conditional 
probability to have a given symbol only depends on the results of the last k 
times, i.e. 

P{s{t)\s{t - 1), s(i - 2), . . .) = P{s{t)\s{t -l),s{t-2),...,s{t- k)) . 

(2.21) 



The Shannon entropy is a measure of the "surprise" the source emitting the 
sequences can reserve to us, since it quantifies the richness (or "complexity") 
of the source. This can be precisely expressed by the first theorem of Shannon- 
McMillan [121] that applies to stationary ergodic sources: 
If N is large enough, the ensemble of A^-long subsequences can be partitioned 
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in two classes, f^i{N) and ^o{N) such that all the words Cn G ^i{N) have 
the same probability P{Cn) ~ exp{—Nhsh) and 

^ P(Cn) ^1 for ^ oo (2.22) 

CNeni{N) 

while 

^ P{Cn)^0 iorN^oo. (2.23) 

CAr6no(iV) 

The meaning of this theorem is the following. An m-states process admits 
in principle possible sequences of length A^, but the number of typical 
sequences Neff{N) (those ones in fli{N)) effectively observable is 

Neff{N) ~ eMNhsh) ■ (2.24) 

Note that A^e// ^ ttT'^ if hsh < Inm. Moreover the entropy per symbol, hsh, 
is a property of the source. Because of the ergodicity it can be obtained by 
analyzing just one single sequence in the ensemble of the typical ones, and 
it can also be viewed as a property of each one of the typical sequences. 
Therefore, as in the following, one may speak about the Shannon entropy of 
a sequence. 

The above theorem in the case of processes without memory is nothing but the 
law of large numbers. Let us observe that (2.24) is somehow the equivalent in 
information theory of the Boltzmann equation in statistical thermodynamics: 
5* oc In W, being W the number of possible microscopic configurations and 5" 
the thermodynamic entropy. This justifies the name "entropy" for hsh- Under 
rather natural assumptions it is possible to prove that the Shannon entropy, 
apart from a multiplicative factor, is the unique quantity which characterizes 
the "surprise" [121]. 

Let us now mention another important result about the Shannon entropy. 
It is not difficult to recognize that the quantity hsh sets the maximum com- 
pression rate of a sequence {s(l), s(2), s(3), . . .}. Indeed a theorem of Shannon 
states that, if the length T of a sequence is large enough, one cannot construct 
another sequence (always using m symbols), from which it is possible to re- 
construct the original one, whose length is smaller than {hsh/ In m)T [201]. In 
other words /i^/j/lnm is the maximum allowed compression rate. 

The relation between Shannon entropy and the data compression problem is 
well highlighted by considering the Shannon-Fano code to map J\f objects (e.g. 
the A^-words Cn) into sequences of binary digits (0, 1) [219]. The main goal 
in building a code is to define the most efficient coding procedure, i.e. the 
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one which generates the shortest possible (coded) sequence. The Shannon- 
Fano code is as follows. At first one orders the M objects according to their 
probabilities, in a decreasing way, Pi,P2, ■ ■ ■ ,PAf- Then the passage from the J\f 
objects to the symbols (0, 1) is obtained by defining the coding E(r), of binary 
length l{E{r)), of the r-th object with the requirement that the expected total 
length, J2rPr^r, be the minimal one. This can be realized with the following 
choice 

-\n2Pr<£{E{7-)) <l-\n2Pr. (2.25) 



In this way highly probable objects are mapped into short code words while 
the low probability ones are mapped to longer code words. So that the average 
code length is bounded by 



^^<^,AEir))<!^^, (2.26) 



and in the limit N ^ oo one has 



lim 



N 



N 



lim 

N^<x 



Y.rPAE{r)) hsh 



N 



In 2 



(2.27) 



i.e., in a good coding, the mean length of a iV-word is equal to N times the 
Shannon entropy (apart from a multiplicative factor, due to the fact that in 
the definition (2.20) of hgh we used the natural logarithm and here we want 
to work with a two symbol code) . 

An alternative coding method, based on variable length words, is due to Ziv 
and Lempel [138]. Remarkably it is very efficient for data compression and 
gives the same asymptotic result of the Shannon-Fano code. 



2.2.2 Again on the Kolmogorov-Sinai entropy 

After the introduction of the Shannon entropy we can give a more precise 
definition of the KS-entropy. Consider a trajectory, x(t), generated by a de- 
terministic system, sampled at the times tj = j r, with j = 1, 2, 3, . . .. Perform 
a finite partition A of the phase space. With the finite number of symbols 
{s}_4 enumerating the cells of the partition, the time-discretized trajectory 
x(tj) determines a sequence {s(l), s(2), s(3), . . .}, whose meaning is clear: 
at the time tj the trajectory is in the cell labeled by s{j). To each subse- 
quence of length • r one can associate a word of length A^: Wj^{A) = 
{s{j), s{j + 1), . . . , s{j + {N — 1))). If the system is ergodic, as we suppose, 
from the frequencies of the words one obtains the probabilities by which one 
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calculates the block entropies Hn{A): 



Hn{A) = - Y1 P{W''{A))lnP{W''{A)). (2.28) 



It is important to note that the probabilities P{W^ {A)), computed by the 
frequencies of l^-^(^) along a trajectory, are essentially dependent on the 
stationary measure selected by the trajectory. This implies a dependence on 
this measure of all the quantities defined below, hxs included. Wc shall always 
understand to consider the natural invariant measure and do not indicate this 
kind of dependence. The entropy per unit time of the trajectory with respect 
to the partition A, h{A), is defined as follows: 



hN{A) = -[Hn+i{A) - Hn(A)] , (2.29) 

T 

h{A) = lim hj^iA) = - lim ^,Hn{A) . (2.30) 

N—^oo T AT— >oo W 

Notice that, for the deterministic systems we are considering, the entropy per 
unit time does not depend on the sampling time r [31]. The KS-cntropy, by 
definition, is the supremum of h{A) over all possible partitions [31,74] 

hKS = sup h{A) . (2.31) 

A 



It is not simple at all to determine hxs according to this definition. A useful 
tool in this respect would be the Kolmogorov-Sinai theorem, by means of which 
one is granted that hxs = h{Q) if ^ is a generating partition. A partition is 
said to be generating if every infinite sequence {s{n)}n=i^...,oo individuates a 
single initial point. However the difficulty now is that, with the exception of 
very simple cases, we do not know how to construct a generating partition. 
We only know that, according to the Krieger theorem [133], there exists a 
generating partition with k elements such that e^'^^ < k < e^'^^ + 1. Then, a 
more tractable way to define hxs is based upon considering the partition Ae 
made up by a grid of cubic cells of edge e, from which one has 

hKS^'^i^h{Ae). (2.32) 



We expect that h{Ae) becomes independent of e when Ae is so fine to be 
"contained" in a generating partition. 

For discrete time maps what has been exposed above is still valid, with r = 1 
(however, Krieger's theorem only applies to invertible maps). 
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The important point to note is that, for a truly stochastic (i.e. non-deterministic) 
system, with continuous states, h{Ae) is not bounded and hxs — oo- 



2.2.3 Algorithmic complexity 

We saw that the Shannon entropy puts a hmit on how efficiently the ensemble 
of the messages emitted by a source can be coded. We may wonder about the 
compressibility properties of a single sequence. This problem can be addressed 
by means of the notion of algorithmic complexity, that is concerned with the 
difficulty in reproducing a given string of symbols. 

Everybody agrees that the binary digits sequence 

0111010001011001011010... (2.33) 

is, in some sense, more random than 

1010101010101010101010... (2.34) 

The notion of algorithmic complexity, independently introduced by Kolmogorov 
[126], Chaitin [53,56] and Solomonov [207], is a way to formalize the intuitive 
idea of randomness of a sequence. 

Consider a binary digit sequence (this does not constitute a limitation) of 
length N, (?'i, • • • , "^jv), generated by a certain computer code on some ma- 
chine Ai. One defines the algorithmic complexity, or algorithmic information 
content, Km{N) of a A^-sequence as the bit length of the shortest computer 
program able to give the A'"-sequence and to stop afterward. Of course, such a 
length depends not only on the sequence but also on the machine. However, 
a result of Kolmogorov [126] proves the existence of a universal computer, U, 
that is able to perform the same computation a program p makes on Ai with 
a modification of p that depends only on M.. This implies that for all finite 
strings: 

Ku{N)<KMiN)+CM, (2.35) 

where Ku{N) is the complexity with respect to the universal computer and 
Cm depends only on the machine Ai. At this point we can consider the algo- 
rithmic complexity with respect to a universal computer - and we can drop the 
machine dependence in the symbol for the algorithmic complexity, K[N). The 
reason is that we are interested in the limit of very long sequences, — > oo. 
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for which one defines the algorithmic complexity per unit symbol: 

C = lim , (2.36) 



that, because of (2.35), is an intrinsic quantity, i.e. independent of the machine. 

Now coming back to the two A^-sequences (2.33) and (2.34), it is obvious that 
the second one can be obtained with a small-length minimal program, i.e. 

N 

"PRINT 10 — TIMES" . (2.37) 



The bit length of the above program is O(lniV) and therefore when taking the 
limit iV ^ oo in (2.36), one obtains C = 0. Of course K{N) cannot exceed 
N, since the sequence can always be obtained with a trivial program (of bit 
length N) 

" PRINT ii, i2,..., ijv" . (2.38) 



Therefore, in the case of a very irregular sequence, e.g., (2.33) one expects 
K{N) oc N i.e. C 7^ 0. In such a case one calls the sequence complex (i.e. of 
non zero algorithmic complexity) or random. 

Algorithmic complexity cannot be computed. Since the algorithm which com- 
putes K{N) cannot have less than K{N) binary digits and since in the case of 
random sequences K(N) is not bounded in the limit N ^ 00 then it cannot 
be computed in the most interesting cases. The un-computability of K{N) 
may be understood in terms of Godel's incompleteness theorem [54-56]. Be- 
yond the problem of whether or not K{N) is computable in a specific case, the 
concept of algorithmic complexity brings an important improvement to clarify 
the vague and intuitive notion of randomness. For a systematic treatment of 
algorithmic complexity, information theory and data compression see [142]. 

There exists a relation between the Shannon entropy, hsh, and the algorithmic 
complexity C. It is possible to show that 

l^(JSm^J_ (2.39) 

N^oo Hn ln2' ^ ^ 



where {K{N)) = Y.Cm P^^n)Kc^{N), being Kcj^{N) the algorithmic com- 
plexity of the iV-words. Therefore the expected complexity {K{N)/N) is 
asymptotically equal to the Shannon entropy (apart the In 2 factor). 
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Eq. (2.39) stems from the results of the Shannon-McMillan theorem about the 
two classes of sequences (i.e. Qi{N) and Qq{N)). Indeed in the limit of very 
large N, the probability to observe a sequence in fli{N) goes to 1, and the 
entropy of such a sequence as well as its algorithmic complexity equals the 
Shannon entropy. Apart from the numerical coincidence of the values of C and 
/i5'/i/ln2 there is a conceptual difference between the information theory and 
the algorithmic complexity theory. The Shannon entropy essentially refers to 
the information content in a statistical sense, i.e. it refers to an ensemble of 
sequences generated by a certain source. On the other hand, the algorithmic 
complexity defines the information content of an individual sequence [96]. 
Of course, as noted above, the fact that it is possible to use probabilistic 
arguments on an individual sequence is a consequence of the ergodicity of the 
system, which allows to assume good statistical properties of arbitrary long 
A^- words. 



For a dynamical system one can define the notion of algorithmic complexity 
of the trajectory starting from the point x, C(x). This requires the introduc- 
tion of finite open coverings of the phase space, the consideration of symbolic 
sequences thus generated, for the given trajectory, sampled at constant time 
intervals, and the searching of the supremum of the algorithmic complexity 
per symbol at varying the coverings [5]. Then it can be shown - Brudno's 
and White's theorems [42,225] - that for almost all x (we always mean with 
respect to the natural invariant measure) one has: 

C(x) = ^ , (2.40) 



where, as before, the factor In 2 is a conversion factor between natural loga- 
rithms and bits. 

This result says that the KS-entropy quantifies not only the richness, or sur- 
prise, of a dynamical system but also the difficulty of describing (almost) 
everyone of its typical sequences. 



2.3 Algorithmic complexity and Lyapunov Exponent 



Summing up, the theorem of Pesin together with those of Brudno and White 
show that a chaotic dynamical system may be seen as a source of messages 

that cannot be described in a concise way, i.e. they are complex. We expose 
here two examples that may help in understanding the previous conclusion 
and the relation between the Lyapunov exponent, the ATS'-entropy and the 
algorithmic complexity. 
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Following Ford [79,80], let us consider the shift map 

x{t + l) = 2x{t) modi. 



(2.41) 



which has A = In 2. If one writes an initial condition in binary representation, 
i.e., x{0) = J2'jLi «j 2"-^, such that Oj = or 1, it is clearly seen that the action 
of the map (2.41) on x{0) is just a shift of the binary coordinates: 

oo oo 

x{l)^J2^j+i'^~' ■■■ x{t) = Y.a,+a~'. (2.42) 

j=i i=i 

With this observation it is possible to verify that K[N) ~ N for almost 
all the solutions of (2.41). Let us consider x{t) with accuracy and a;(0) 
with accuracy 2~', of course I — t + k. This means that, in order to obtain 
the k binary digits of the output solution of (2.41), we must use a program 
of length no less than I = t + k. Basically one has to specify 0,1,0,2, . . . ,ai. 
Therefore wc arc faced with the problem of the algorithmic complexity of the 
binary sequence (oi, 02, ... , Ooo) which determines the initial condition x{0). 
Martin-Lof [156] proved a remarkable theorem stating that, with respect to 
the Lebesgue measure, almost all the binary sequences (oi, 02, ... , Ooo), which 
represent a real number in [0, 1], have maximum complexity, i.e. K{N) ~ N. In 
practice no human being will ever be able to distinguish the typical sequence 
(oi, 02, . . . , Ooo) from the output of a fair coin toss. 

Finally let us consider a. Id chaotic map 

x{t + l) = f{x{t)). (2.43) 

If one wants to transmit to a friend on Mars the sequence {x{t), t = 1, 2, . . . , T} 
accepting only errors smaller than a tolerance A, one can use the following 
strategy [174]: 

(1) Transmit the rule (2.43): for this task one has to use a number of bits 
independent of the length of the sequence T. 

(2) Specify the initial condition a;(0) with a precision Sq using a finite number 
of bits which is independent of T. 

(3) Let the system evolve till the first time ri such that the distance between 
two trajectories, that was initially 6x{0) = 60, equals A and then specify 
again the new initial condition x{ti) with precision Sq. 

(4) Let the system evolve and repeat the procedure (2-3), i.e. each time the 
error acceptance tolerance is reached specify the initial conditions, x{ti + 
r2), x{ri + r2 + Ta) . . ., with precision Sq. The times Ti,T2, ■ ■ ■ are defined 
as follows: putting x (ti) = x{ti) +60, T2 is given by the minimum time 
such that \x (ri + T2) — x{ti + T2)| > A and so on. 
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By following the steps (1 — 4) the friend on Mars can reconstruct with a 
precision A the sequence {x{t)} simply iterating on a computer the system 
(2.43) between 1 and Ti — 1, ri and Ti + T2 — I, and so on. 

Let us now compute the amount of bits necessary to implement the above 
transmission (1-4). For simplicity of notation we introduce the quantities 

-fi ^ -In ^ (2.44) 

n do 



which can be considered as a sort of effective Lyapunov exponents (see Sect. 3.1). 
The LE A can be written in terms of {71} as follows 

A = (7.) = ^ = =ln^ (2.45) 



where 

is the average time after which we have to transmit the new initial condition 
(let us observe that to obtain A from the 7i's one has to perform the average 
(2.45) because the transmission time, r-j, is not constant). If T is large enough 
the number of transmissions, A^, is T/r ~ AT/ln(A/5o)- Therefore, noting 
that in each transmission for the reduction of the error from A to 5o one needs 
to use ln2(A/(5o) bits, the total amount of bits used in the whole transmission 
is 

T A A 

lln2_ = _T. (2.46) 
T 5o ln2 ^ ' 



In other words the number of bits for unit time is proportional to A. 

In more than one dimension, we have simply to replace A with Hks in (2.46). 
To intuitively understand this point one has to repeat the above transmission 
procedure in each of the expanding directions. 

In this section, we briefly discussed how the limit of predictability, the fact 
that a sequence cannot be arbitrarily compressed, and the information con- 
tained in a signal are deeply related. In the following we will mainly discuss 
the dynamical system point of view, i.e., in terms of Lyapunov exponents, 
Kolmogorov Sinai entropy and their generalizations for less ideal cases. 
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3 Limits of the Lyapunov exponent for predictability 



We saw how a first obvious way for quantifying the predictabihty of a physical 
system is in terms of the predictability time Tp, i.e. the time interval on which 
one can typically forecast the system. A simple argument previously suggested 



However, in any realistic system, relation (3.1) is too naive to be of actual rel- 
evance. Indeed, it does not take into account some basic features of dynamical 
systems: 

- The Lyapunov exponent (2.9) is a global quantity: it measures the average 
rate of divergence of nearby trajectories. In general there exist finite-time 
fluctuations and the probability distribution functions (pdf ) of these fluctua- 
tions is important for the characterization of predictability. The generalized 
Lyapunov exponents have been introduced with the purpose to take into 
account these fluctuations [23,85]. 

- The Lyapunov exponent is defined for the linearized dynamics, i.e., by 
computing the rate of separation of two infinitesimally close trajectories. On 
the other hand, for measuring the predictability time (3.1) one is interested 
in a finite tolerance A, because the initial error 6o is typically finite. A 
recent generalization of the Lyapunov exponent to finite size errors extends 
the study of the perturbation growth to the nonlinear regime, i.e. both 5o 
and A are not infinitesimal [11,12]. 



3.1 Characterization of finite-time fluctuations 

Let us consider the linear response, at a delay to an infinitesimal perturba- 
tion (5x(0): 



from which the LE is computed according to (2.9). In order to take into account 
the finite-time fiuctuations, we can compute the different moments {R{ty) and 
introduce the so-called generahzed Lyapunov exponents (of order q) [23,85]: 





\s^{t)\ 
MO) I' 



(3.2) 



L{q)^\]m^ln{R{ty) 



(3.3) 
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where (...) indicates the time average along the trajectory (see Sect. 2). It is 
easy to show that 



Ai 



dL{q) 
dq 



(3.4) 



q=0 



In the absence of fluctuations, Ai completely characterizes the error growth 
and we have L{q) — Xiq, while in the general case L{q) is concave in q [172]. 
Before discussing the properties of the generalized Lyapunov exponents, let 
us consider a simple example with a non trivial L[q). The model is the one- 
dimensional map 

, forO<x<a 
4^+1)= ^ - (3.5) 

V 1—a — 



which for a = 1/2 reduces to the tent map. For a ^ 1/2 the system is charac- 
terized by two different growth rates. The presence of different growth rates 
makes L{q) non-linear in q. Since the map (3.5) is piecewise linear and with a 
uniform invariant density, the explicit computation of L{q) is very easy. The 
moments of the response after a time t are simply given by 



{Rity 



(3.6) 



From (3.3) and (3.6) we have: 

L{q) = In fa^-« + (1 - a)^-« 



(3.7) 



which recovers the non intermittent limit L{q) = gin 2 in the symmetric case 
a — 1/2. In the general case, assuming 0<a<l/2, we have that for q — > 
-|-oo, L{q) is dominated by the less probable, most unstable contributions 
and L{q)/q ~ — ln(a). In the opposite limit, q — > — oo, we obtain L{q)/q ~ 
-ln(l - a). 

We now show how L{q) is related to the fluctuations of R{t) at flnite time t. 
Deflne an "effective" Lyapunov exponent ^{t) at time t by 

R{t) - e^(*)* . (3.8) 



In the limit t ^ oo, the Oseledec theorem [169] assures that, for typical 
trajectories, 7(t) = Ai = —a In a — (1 — a) ln(l — a). Therefore, for large t, the 
probability density of ^{t) is peaked at the most probable value Ai. Let us 
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introduce the probability density Pti'j) of observing a given 7 on a trajectory 
of length t. Large deviation theory suggests 



P*(7)-e-^W*, (3.9) 



where 15(7) is the Cramer function [216]. The Oseledcc theorem implies that 
limj^oo Pt{l) = ^{1 ~ ^i), this gives a constraint on the Cramer function, i.e. 
5(7 = Ai) = and 5(7) > for 7 ^ Ai. 

The Cramer function S{'~f) is related to the generalized Lyapunov exponent 
L{q) trough a Legendre transform. Indeed, at large t, one has 

{R{ty) = I rf7^t(7)e^^* ~ e^(«)* , (3.10) 



by a steepest descent estimation one obtains 

L(?)= max [97-5(7)] . (3.11) 

In other words each value of q selects a particular j*{q) given by 



= q. (3.12) 

7* 



We have already discussed that, for negligible fluctuations of the "effective" 
Lyapunov exponents, the LE completely characterizes the error growth and 
L{q) — Xiq. In presence of fluctuations, the probability distribution for R{t) 
can be approximated by a log-normal distribution. This can be understood 
assuming weak correlations in the response function so that (3.2) factorizes in 
several independent contributions and the central limit theorem applies. We 
can thus write 

r,/r,N 1 f (lni?-Alt)2\ 



where Ai and /i are given by 



Ai= lim - {In R(t)) 

t—fOC f 



1^ ^ lim J ({In R{ty) - {In R{t))^) . (3.14) 
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The log-normal distribution for R corresponds to a Gaussian distribution for 

7 

.W^<1^. (3.15) 



and to a quadratic in q generalized Lyapunov exponent: 

L{q) = Aig + i/xg^ . 



(3.16) 



Let us remark that, in general, the log-normal distribution (3.13) is a good 
approximation for non extreme events, i.e. small fluctuation of 7 around Ai, 
so that the expression (3.16) is correct only for small q (see Fig. 1). This is 
because the moments of the log-normal distribution grow too fast with q [168]. 
Indeed from (3.12) we have that the selected ■j*{q) is given by j*{q) = \i + fiq 
and thus 7*(g) is not finite for q 00. This is unphysical because 7* (00) is 
the fastest error growth rate in the system and, we may reasonably suppose 
that it is finite. 

Let us consider again the map (3.5). In this case we have Ai = L'{0) ~ 
—a ln(a) — (1 — a) ln(l — a) and fi = L"{0) = a(l — a) (ln(a) — ln(l — a))^, 
which are nothing but the coefficients of the Taylor expansion of (3.7) around 
g = 0. For large q the log-normal approximation gives L{q)/q ~ qn/2 while 
the correct limit is the constant L{q)/q ~ — ln(a). 




Fig. 1. Generalized Lyapunov exponent, L{q) for the map (3.5) with a = 0.3 (solid 
line) compared with the linear non intermittent approximation, Xiq (dashed line), 
and with the log-normal one Eq. (3.16) (dotted line). 

Nevertheless, Ai and /i are the two basic parameters for characterizing the 
intermittency of a system. To be more specific, let us remark that Pt{R) (3.13) 
reaches its maximum for 



Rmax (^) 6 



(Ai-/i)t 



(3.17) 
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so that for i — > oo: 

Rmax ^ OO if 11/ Xi < 1 



Rmax^O if///Ai>l. ^^'^^^ 



Thus in the weak intermittency hmit, n/Xi < 1, the most probable response 
function Rmax{t) follows the correct behavior (with the corrected exponent 
Ai — /x). In the strong intermittent limit, n/Xi > 1, the most probable estima- 
tion breaks down because it predicts an asymptotic stable phase Rmax{t) — 
instead of the chaotic exponential growth. 

As in the case of the first LE, it is possible to introduce higher order general- 
ized Lyapunov exponents. Starting from the n-order response function 
(2.10), we define 

L(")(g) = lim - ln(i?(")(i)«) , (3.19) 

t-»oo t 



where L'^^^q) — L{q). Prom (2.11) we have 

" . _ dLW(g) 

1 



(3.20) 

q=0 



The generalized L^"'\q) represents the fluctuations of the exponential diver- 
gence of a n-dimensional volume in phase space [172]. The properties of L("^)(g) 
are analogous to the properties of L{q), i.e. L^'^\q) is a concave function of q 
for any n and for a non-intermittent behavior they are linear in q. 



3.2 Renyi entropies 



In Section 2.1.2 we defined the Kolmogorov-Sinai entropy (2.13) and discussed 
its relation with the Lyapunov exponents by means of the Pesin relation (2.16). 
Analogously to the generahzed LE, it is possible to introduce a generalization 
of the Kolmogorov-Sinai entropy in order to take into account the intermit- 
tency. 

Let us recall the definition of Kolmogorov-Sinai entropy 

fe = -limlimjim ^ P(iy^(A)) lnP(iy^(A)) (3.21) 



where Ae is a partition of the phase space in cells of size e and W^{Ae) 
indicates a sequence of length N in this partition. The generalized Renyi 
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entropies [171,172], Kq, can be introduced by observing that (3.21) is nothing 
but the average of — lnP(H^^) with the probabihty P{W^): 



Kg^- hmhm hm -— -In V P(W^(Ae)y ■ (3.22) 



As in (3.4) one has h^s = hm^^i Kg = Ki; in addition from general results of 
probability theory, one can show that Kg is monotonically decreasing with q. 

It will not be surprising that the generalized Renyi entropies are related to 
the generalized Lyapunov exponents L{q). Introducing the number of non- 
negative Lyapunov exponents n* (i.e. A„* > 0, A„*+i < 0), the Pesin relation 
(2.16) can be written as 



li'KS — / . At — 



dq 

Moreover, one has [171]: 



(3.23) 



g=0 



= ^ . (3.24) 



-5 



3.3 The effects of intermittency on predictability 



We have seen that intermittency can be described, at least at a qualitative 
level, in terms of Ai and /i, which are the two parameters characterizing the 
log-normal approximation. We discuss now the relevance of the log-normal 
approximation for the predictability time Tp. 

The predictability time Tp is defined as the time it takes for the error of initial 
size Sq to grow up to the tolerance A 



Tp — min 



A- 

t : R{t) > - 



(3.25) 



In the framework of the log-normal approximation, we can write 

In R{t) = Xit + y/JIw{t) , (3.26) 

where w{t) is a Wiener process with w{0) = 0, {w{t)) = and {w{t)w{t')) ~ 
min(t, t'). In this case the computation of Tp is reduced to a first exit problem. 
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which is well known in stochastic process theory [45,78]. The solution gives 
the pdf of [66]: 



HA/6,) 



(All; - ln(A/5o))^ 



(3.27) 



Notice that the (3.27) is not normalized, since there is a finite probability of 
"no exit" . Of course, this is an artifact of the approximation in terms of the 
stochastic process (3.26). In non pathological chaotic systems one expects that 
p{Tp) is normalized. 
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Fig. 2. Rescaled pdf, p(Tp)a, of the predictability time Tp for the Lorenz model 
(3.29): (a) with r = 28 (weak intermittency) the average predictability time is 
(Tp) = 10.84 and its variance is cr^ = 3.12 while A = 0.90, /i = 0.06 ± 0.01; (b) with 
r = 166.3 (strong intermittency) and (Tp) = 8.2385 and = 19.75, while A = 1.18 
and /i = 3.9 it 1. The dashed line is the Gaussian distribution. 

In the limit of weak intermittency /i/Ai ^ 1, p{Tp) is almost Gaussian and the 
mean value (Tp) is close to the most probable value of (3.27) corresponding to 
the naive estimation (3.1). On the contrary, in the strong intermittent limit. 
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A*/Ai ^ 1, the pdf of Tp shows an asymmetric "triangular shape" and the 
most probable value is 



1 



ln(A/5o)' . 



(3.28) 




3/x 



In order to see the effects of intcrmittency on the predictability time, let us 
consider as an example the Lorenz system [145]: 



with the standard values a — 10 and b — 8/3. For r = 28, the Lorenz model 
is very weakly intermittent, ///A ~ 7 x 10~^, and the pdf of the predictabihty 
time is very close to a Gaussian (Fig. 2). On the contrary, for r = 166.3 the 
Lorenz model becomes strongly intermittent [191], fi/\ ~ 3.3 and the pdf 
of the predictability time displays a long exponential tail responsible for the 
deviation from (3.1). 

This qualitative behavior is typical of intermittent systems. In Section 5.3 we 
will see a more complex example in the context of turbulence. 

3.4 Growth of non infinitesimal perturbations 

In realistic situations, the initial condition of a system is known with a limited 

accuracy. In this case the Lyapunov exponent is of little relevance for the 
characterization of predictability and new indicators are needed. To clarify 
the problem, let us consider the following coupled map model: 



where x G IR^, y E IR^, R is a rotation matrix of arbitrary angle ^, h is a 
vector function and G is a chaotic map. For simplicity we consider a linear 
coupling h{y) = {y,y) and the logistic map G{y) = 4y{l — y). 

For £ = wc have two independent systems: a regular and a chaotic one. 
Thus the Lyapunov exponent of the x subsystem is Xx{e = 0) = 0, i.e., it 
is completely predictable. On the contrary, the y subsystem is chaotic with 
X = Ai =ln2. 




(3.29) 




(3.30) 
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If we now switch on the (small) coupling [e > 0) we are confronted with 
a single three-dimensional chaotic system with a positive global Lyapunov 
exponent 

X^Xy + 0(s). (3.31) 

A direct application of (3.1) would give 

r^(^) ~ ~ f , (3.32) 

Ay 

but this result is clearly unacceptable: the predictability time for x seems to be 
independent of the value of the coupling e. Let us underline that this is not due 
to an artifact of the chosen example. Indeed, one can use the same argument 
in many physical situations [32]. A well known example is the gravitational 
three body problem with one body (asteroid) much smaller than the other two 
(planets). If one neglects the gravitational feedback of the asteroid on the two 
planets (restricted problem) one has a chaotic asteroid in the regular field of 
the planets. As soon as the feedback is taken into account (i.e. £ > in the 
example) one has a non-separable three body system with a positive LE. Of 
course, intuition correctly suggests that it should be possible to forecast the 
motion of the planets for very long times if the asteroid has a very small mass 
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Fig. 3. Growth of error |5x(f)| for the coupled map (3.30). The rotation angle is 

9 = 0.82099, the coupling strength e = 10~^ and the initial error only on the y 
variable is Sy = 5o = lO"^'^. Dashed line |5x(t)| ^ e^^* where Ai = ln2, solid line 

|(^x(i)| ~iV2. 

The apparent paradox arises from the use of (3.1), which is valid only for the 
tangent vectors, also in the non infinitesimal regime. As soon as the errors 
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become large one has to take into account the full nonlinear evolution. The 
effect is shown for the model (3.30) in Figure 3. The evolution of (5x is given 

by 

Sx{t + 1) = RSx{t) + e5h{y) , (3.33) 

where, with our choice, 5h. — (5y,6y). At the beginning, both |5x| and 5y 
grow exponentially. However, the available phase space for y is finite and the 
uncertainty reaches the saturation value 6y ~ 0(1) in a time t ~ 1/Ai. At 
larger times the two realizations of the y variable are completely uncorrelated 
and their difference 6y in (3.33) acts as a noisy term. As a consequence, the 
growth of the uncertainty on x becomes diffusive with a diffusion coefficient 
proportional to £^ [32] 

|5x(i)| (3.34) 

so that: 

~ . (3.35) 

This example shows that, even in simple systems, the Lyapunov exponent can 
be of httle relevance for the characterization of the predictability. 

In more complex systems, in which different scales are present, one is typically 
interested in forecasting the large scale motion, while the LE is related to the 
small scale dynamics. A familiar example is weather forecast: the LE of the 
atmosphere is indeed rather large due to the small scale convective motion, but 
(large scale) weather prediction is possible for about 10 days [146,160]. It is 
thus natural to seek for a generahzation of the LE to finite perturbations from 
which one can obtain a more realistic estimation for the predictability time. 
It is worth underlining the important fact that finite errors are not confined 
in the tangent space but are governed by the complete nonlinear dynamics. In 
this sense the extension of the LE to finite errors will give more information 
on the system. 

Aiming to generalize the LE to non infinitesimal perturbations let us now 
define the Finite Size Lyapunov Exponent (FSLE) [11,12] (see Appendix A 
for the computational details). Consider a reference trajectory, x(t), and a 
perturbed one, x'(t), such that |x'(0) — x(0)| = S (|...| is the Euclidean 
norm but one can also consider other norms). One integrates the two trajecto- 
ries and computes the time Ti{6,r) necessary for the separation |x (t) — x(t)| 
to grow from 6 to r6. At time t = Ti{6,r) the distance between the trajec- 
tories is rescaled to 6 and the procedure is repeated in order to compute 
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The threshold ratio r must be r > 1, but not too large in order to avoid 
contributions from different scales in r(5, r). A typical choice is r = 2 (for 
which t(5, r) is properly a "doubling" time) or r = ^J2. In the same spirit of 
the discussion leading to Eq.s (2.44) and (2.45), we may introduce an effective 
finite size growth rate: 

After having performed M error-doubhng experiments, we can define the FSLE 
as 

A(5) = (7(5,r)), = /--^\ lnr = --l--lnr, (3.37) 



where (t(5, r))e is 



(r(5,r))e = -J.Er„(5,r). (3.38) 

n=l 



(see Appendix A and [12] for details). In the infinitesimal limit, the FSLE 
reduces to the Lyapunov exponent 

limA((5) = Ai. (3.39) 



In practice this limit means that A(5) displays a constant plateau at Ai for 
sufficiently small b (Fig. 3). For finite value of 5 the behavior of A(5) depends 
on the details of the non linear dynamics. For example, in the model (3.30) 
the diffusive behavior (3.34), by simple dimensional arguments, corresponds 
to ~ Since the FSLE measures the rate of divergence of trajectories 
at finite errors, one might wonder whether it is just another way to look at the 
average response (ln(i?(t))) (3.2) as a function of time. A moment of refiection 
shows that this is not the case. Indeed taking the average at fixed time is not 
the same as computing the average doubling time at fixed scale, as in (3.37). 
This is particularly clear in the case of strongly intermittent system, in which 
R{t) can be very different in each realization. In presence of intermittency 
averaging over different realizations at fixed times can produce a spurious 
regime due to the superposition of exponential and diffusive contributions by 
different samples at the same time [10]. 

The FSLE method can be easily applied for data analysis [35]. 

For other approaches to address the problem of non-infinitesimal perturbations 
see [73,212,115]. 
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Fig. 4. A((5) as a function of S for the coupled map (3.30) with e = 10'^. The 
perturbation has been initiahzed as in Fig. 3. For 5—^0, X{S) 2± Ai (soUd hne). The 
dashed hne shows the behavior X{d) ^ 

3.5 The e-entropy 

The Kolmogorov-Sinai entropy, hxs (2-13), of a system measures the amount 
of information per unit time necessary to record without ambiguity a generic 
trajectory of a system. Since the computation of hxs involves the limit of 
arbitrary fine resolution and infinite times (see Sect. 2.1.2), it turns out that, 
practically, for most systems it is not possible to compute hxs- Nevertheless, in 
the same philosophy of the FSLE, by relaxing the strict requirement of repro- 
ducing a trajectory with arbitrary accuracy, one can introduce the e-entropy 
which measures the amount of information for reproducing a trajectory with 
accuracy e in phase-space. Roughly speaking the e-cntropy can be considered 
the counterpart, in information theory, of the FSLE (as the T^TS'-entropy is for 
the Lyapunov exponent). Such a quantity was originally introduced by Shan- 
non [201], and by Kolmogorov [124]. Recently Gaspard and Wang [89] made 
use of this concept to characterize a large variety of processes. 

We start with a continuous (in time) variable x(t) G IR'^, which represents 
the state of a rf-dimensional system, wc discrctized the time by introducing 
an interval r and we consider the new variable 

X^'"\t) = (x(i), x(i + r), . . . , x(i + (m - 1)t)) . (3.40) 

Of course 'X.^"^\t) e H'"'' and it corresponds to the trajectory which lasts for 
a time T — mr. 
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In data analysis, the space where the state vectors of the system Uvc is not 
known. Moreover, usually only a scalar variable u(t) can be measured. In such 
a case, one considers vectors {u{t),u{t + t), . . . ,u{t + mr — r)), that live in 
H"* and allow a reconstruction of the original phase space, known as delay 
embedding in the literature [209,199] (see also [1,2,114,170]), and it is a special 
case of (3.40). 

Introduce now a partition of the phase space IR'^, using cells of edge e in each 
of the d directions. Since the region where a bounded motion evolves contains 
a finite number of cells, each 'K^'^\t) can be coded into a word of length m, 
out of a finite alphabet: 

XM(i) — > W^ie, t) = {i{e, t),i{e, t + r),..., i{e, t + mr-r)), (3.41) 

where i{e,t + jr) labels the cell in H'^ containing x(t + jr). Prom the time 
evolution of X(™)(t) one obtains, under the hypothesis of ergodicity, the prob- 
abilities P(iy"(e)) of the admissible words {iy"(e)}. We can now introduce 
the (e, T)-entropy per unit time, /t(e, r) [201]: 

hmie, t) = -[H^+i{e, r) - H^{e, r)] (3.42) 

T 

1 1 

h{e,T)= lim /i„(e,T) = - lim — i7„(e,T), (3.43) 

m— ►oo J- m-+oo 777, 

where is the block entropy of block length m: 

Hra{e,T) = - P{W^{e))hiP{W"\e)). (3.44) 

{W"(e)} 



For the sake of simphcity, we ignored the dependence on details of the par- 
tition. To make h{e, r) partition-independent one has to consider a generic 
partition of the phase space {A\ and to evaluate the Shannon entropy on 
this partition: /i5/j(^, r). The e-entropy is thus defined as the infimum over 
all partitions for which the diameter of each cell is less than e [89]: 

/i(£,t)= ^^.inf hshiAr). (3.45) 



Note that the time dependence in (3.45) is trivial for deterministic systems, 
and that in the limit e — > one recovers the Kolmogorov- Sinai entropy 

hxs — liin /i(e, r). 



The above entropies i/m(e) have been introduced by using a partition and the 
usual Shannon entropy; however it is possible to arrive at the same notion. 
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starting from other entropy-likc quantities, that are numerically more conve- 
nient. For example Cohen and Procaccia [61] proposed to estimate H„i{e) as 
follows. Given a signal composed of N successive records and the embedding 
variable X^"*^ let us introduce the quantities: 

) = ^^T.e{e - \X^-\iT) - XM(^v)l) , (3.46) 



then the block entropy HyJ^e) is given by 

- - (^_^^i) Elnnr(e) . (3.47) 



In practice (e) is an approximation of P(M^™'(e)). From a numerical point 
of view, correlation entropies [95,210] are sometimes more convenient, so that 
one studies 

^i^H^) = -In [ ^_\^^ Y.nf\e)\ < Hi^\e) . (3.48) 



This corresponds to approximate the Shannon by the Renyi entropy of order 
g = 2 [114]. 

The (e, r)-entropy h{e, r) is well defined also for stochastic processes. Actu- 
ally the dependence of h{e, r) on e can give some insight into the underlying 

stochastic process [89], for instance, in the case of a stationary Gaussian pro- 
cess with spectrum S{u!) cx u!~^ one has [124]: 

limMe,T)-^. (3.49) 



However, we have to stress that the behavior predicted by Eq. (3.49) may be 
difficult to be experimentally observed mainly due to problems related to the 
choice of r [51,3] (see also Appendix C). 



4 Predictability in extended systems 



Here we consider extended dynamical systems, whose degrees of freedom de- 
pend on space and time, and which can display unpredictable behaviors both 
in the time and space evolution, i.e. spatio-temporal chaos. The inadequacy 
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of the Lyapunov exponents in characterizing predictabihty becomes now well 
evident. 

Following Hohenberg and Shraiman [104] (see also [68]) we give a more pre- 
cise meaning to the terms spatio-temporal chaos and extended systems. For a 
generic system of size L, we can define three characteristic lengths: in, £e, 
^ respectively associated to the scales of dissipation (i.e. the scale at which 
dissipation becomes effective, smaller scales can be considered as inactive), 
excitation (i.e. the scale at which energy is injected in the system) and cor- 
relation (that we assume can be suitably defined). Now one has two limiting 
situations. 

When all the characteristic lengths are of the same order {iD,iE,C ^ 0{L)) 
distant regions of the system are strongly correlated. Because of the coherence, 
the spatial nature is not very important and one speaks of temporal chaos, i.e. 
the system is basically low dimensional. 

When L 3> 3> distant parts of the system are weakly correlated so that 
the number of (active) degrees of freedom and consequently the number of 
positive Lyapunov exponents, the Kolmogorov-Sinai entropy and the attractor 
dimension, Dp-, are extensive quantities, i.e. they increase with the system size, 
L. Here, spatial aspects are crucial and one speaks of spatio-temporal chaos, 
e.g., Rayleigh-Bernad convection for large aspect ratio [153]. 

The above picture is just an approximative scenario (see [104] for further 
details) but sufficiently broad to include systems ranging from fluid dynamics 
to biological and chemical reactions [68,153]. In spite of the fact that turbulent 
flows fit in this broad definition we shall discuss the predictability problem in 
turbulence in the next Section. 

For detailed discussions on different physical and chemical systems which can 
be included in the above definition see [68,38]. Here we discuss the predictabil- 
ity problem in a restricted class of models, which are relatively simple from 
a computational and theoretical point of view but, nevertheless, possess the 
essential phenomenology of spatio-temporal chaos. 



4.1 Simplified models for extended systems and the thermodynamic limit 

A great simplification in the study of extended systems, usually described by 
partial differential equations (PDE), can be achieved by considering discrete 
time and space models, and introducing the Coupled Map Lattices (CML) 
[108], i.e. maps defined on a discrete lattice. A typical 1-dimensional CML (the 
extension to d-dimensions is straightforward) can be written in the following 
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way: 



Mt)] + 7^ E £,-(fa[x.+,-(t)]+f„[x,_,-(i)]) , (4.1) 



L/2 



X,(i + l) = (l-£o)fa[: 



j=-Z,/2 



with Eq = J2j=i L is the lattice size, i — —L/2, .... L/2, x G IR" is the state 
variable which depends on the site and time, and fa G IR" IR" is a non linear 
map, which drives the local dynamics and depends on a control parameter a. 
Usually, periodic boundary conditions ^i+L = Xj are assumed and, for scalar 
variables {n — 1), one studies coupled logistic maps, fa{x) — ax{l — x) or tent 
maps, fa{x) = a\l/2 - \x - l/2\\. 

The parameters {si} rule the strength and the form of the coupling and they 
are chosen according to the physical system under investigation. For example, 
with Ej — for j > 2, i.e. nearest neighbor coupling, one can mimic PDE's 
describing reaction diffusion processes (indeed formally the equation assumes 
the structure of a discrete Laplacian). However, it could be misleading to con- 
sider CML's simply as discrete approximation of PDE's. Indeed, since the local 
map fa is usually chaotic, chaos in CML, differently from PDE, is the result 
of many interacting chaotic sub-systems. Hence, the correspondence between 
the instability mechanisms in the two type of models is not straightforward 



Other kinds of coupling can be considered to mimic different physical situa- 
tions, e.g., asymmetric CML (see Sect. 4.5) for studying convective instabilities 
[106,39,217], or mean field (globally coupled maps) version of (4.1) (Sect. 4.3) 
for studying neural network or population dynamics [118]. Further generaliza- 
tions are quoted in Ref. [112]. 

Lyapunov exponents, attractor dimensions and entropies can be defined (and, 
at least the Lyapunov exponents, numerically computed) also for extended 
systems. In particular, for L < oo the CMLs have finite dimensional phase 
space and the above quantities are well defined. In PDE's some difficulties can 
rise due to the problem of the non equivalence of the norms [127] : Lyapunov 
exponents and consequently the characterization of the predictability may 
depend on the chosen norm. We shall see in Sect. 4.8 that this is not just a 
subtle mathematical problem. 

In order to build a statistical description of spatio-temporal chaos, as Ruelle 
pointed out [197], one has to require the existence of a good thermodynamic 
limit for the Lyapunov spectrum {Aj(L)}j=i i. This means the existence of the 
limit 



[68]. 



lim Xi{L) = A{x) , 



(4.2) 
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where x = i/L is a, continuous index in [0, 1], and A(a;) is a non increasing 
function. The function A(a;) can be viewed as a (iensit?/ of Lyapunov exponents. 
If such hmit does not exist, the possibihty to build a statistical description 
of spatio-temporal chaos would be hopeless, i.e., the phenomenology of these 
systems would depend on L. 

Once the existence of a Lyapunov density is proved, one can generalize some 
results of low dimensional systems [97,44], namely the Kaplan- Yorke conjec- 
ture [117] and the Pesin relation (2.16). For instance, one can generalize (2.16) 
to 

h ]■ 

Hks = lim = / dxK{x)e{K{x)) (4.3) 



being 9{x) the step function. In the same way one can suppose the existence of 
a dimension density Dp, that is to say a density of active degrees of freedom, 
i.e. Vp — lim^^oo Dp/L which by the Kaplan- Yorke [117] conjecture is given 
by [97]: 

Vp 

J dxA{x) = . (4.4) 



The existence of a good thermodynamic limit is supported by numerical sim- 
ulations [109,144] and some exact results [205]. Recently Eckmann and Collet 
[62] have proved the existence of a density of degrees of freedom in the com- 
plex Ginzburg-Landau equation. See also Refs. [97,44] and references therein 
for a discussion on such a problem. 

4-2 Overview on the predictability problem in extended systems 

In low dimensional systems, no matter how the initial disturbance is chosen, 
after a - usually short - relaxation time, Tr, the eigendirection with the largest 
growth rate dominates for almost all the initial conditions (this, e.g., helps in 
the numerical estimates of the Lyapunov exponents [20]). On the contrary, 
in high-dimensional systems this may not be true [92,183,173,186]. Indeed, 
in systems with many degrees of freedom there is room for several choices of 
the initial perturbation according to the specific problem under investigation 
(e.g., locahzed on certain degrees of freedom or homogeneous in all the degrees 
of freedom) , and it is not obvious that for all of them the time Tr needed to 
align along the maximally expanding direction is the same. 

In general the situation can be very complicated. For instance, it is known 
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that, also considering initially homogeneous disturbances, the Lyapunov vec- 
tors can localize (permanently or slowly wandering) on certain degrees of free- 
dom [109,75,186]. Of course, this will severely affect the prediction of the future 
evolution of the system. Indeed, regions of large predictabihty time could co- 
exist with regions of relatively short predictability time. In Ref. [109,112,186] 
one finds an abundance of examples displaying this phenomenology. A detailed 
analysis of this problem is far from the aims of this review; we just mention 
that the behavior of the Lyapunov vectors can range from a strongly localized 
regime (the origin of which can be understood by the analogy with Ander- 
son localization of the wave function in disordered potential [91]) to localized 
wandering structures. In particular, in the latter case there is strong numer- 
ical evidence [185,186] that for a large class of (1-dimensional) systems the 
dynamics of the Lyapunov vectors (actually the logarithm of them) falls into 
the universality class of the 1-dimensional KPZ equation [120] . 

In these situations the main contribution to the predictability time comes from 
the time needed for the perturbation to propagate through the system or to 
align along the maximally expanding direction, which can be of the order of 
the system size [173,183,186]. As a consequence the predictability time can be 
much longer than the rough estimation Tp ~ 1/ A. 

Moreover, the LE can also be unsatisfactory if one is interested in pertur- 
bations with particular space-time shapes. Indeed, these features have lead 
to the introduction of a number of new indicators; for instance, the temporal 
(or specific) Lyapunov exponents [189,139], the spatial Lyapunov exponents 
[91.139] (which characterize respectively perturbations exponentially shaped 
in time and space) or the comoving Lyapunov exponents [71] for the charac- 
terization of the spatio-temporal evolution of localized perturbation [109] and 
of the convective instabilities [8,39]. 

Convectively unstable systems are rather interesting because, even if the LE 
(computed in the stationary frame) is negative, some features of the motion 
can be highly unpredictable (see Sect. 4.6). It is also worth mentioning the 
existence of systems with exponentially long (in the system size) transients 
during which the dynamics is essentially unpredictable despite the fact that 
the LE is negative [69]. This phenomenon, known under the name of stable 
chaos [188], will be briefiy discussed in Sect. 7.3. 

In high-dimensional systems one is also interested in predicting the behavior 
of some average observable to achieve a macroscopic description of the system. 
The coarse grained (hydrodynamic like) dynamics may be non trivial, and the 
largest LE, which is related to the fine grained dynamics, is not relevant for 
characterizing the predictability at a coarse grained level (see Sect. 4.7). 

We conclude this brief overview by mentioning another interesting feature: 
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in spatially distributed systems coherent structures may appear. They move 
maintaining for rather long times their shape. In different contexts one can be 
interested in predicting the evolution of such structures; e.g., cyclonic/anti- 
cyclonic structures in the atmosphere. A reasonable question could be the 
prediction of the center and orientation of such structures: limiting to these 
aspects one can indeed hope to have a rather long predictability time compared 
to the rough estimate Tp ~ 0(1/A). However, since usually such phenomena 
arise in fields, whose evolution is ruled by PDE, the non equivalence of the 
norms makes a general approach to the problem unfeasible. Therefore, one 
has to resort to ad hoc treatments, based on physical intuition to identify the 
most suitable norm to be used for the particular needs (see Sect. 4.8). 

4-3 Butterfly effect in coupled map lattices 

In spatially extended systems it is important to understand the way an uncer- 
tainty initially localized in some region will spread. Here we study in particular 
the time needed for a perturbation, initially seeded in the central site of a lat- 
tice of coupled maps, to reach a prcassigncd value at the border of the lattice 
[173] (see also [109,212,213] and Sect. 4.6). In other terms we wonder about 
the "butterfly effect" starting from the center of the lattice and arriving up 
to the boundary. 

We shall discuss the properties of such time by varying the coupling range from 
local to non local in the 1-dimensional CML (4.1) with periodic boundary 
conditions. We consider two cases: local coupling, i.e. ej = if j > 2, and 
non-local coupling, e.g. 

C2 

£1 = Ci and Ej — — for j > 2 (4.5) 

where a measures the strength of non-locality. The initial perturbation is on 
the central site, i.e. 

\Sxi{0)\ = SoSi,o. (4.6) 

We look at the predictability time Tp needed for the perturbation to reach a 
certain threshold 6max on the boundary of the lattice, i.e. the maximum time, 

t, such that \SxL/2{t)\ < ^max- 

For nearest neighbor coupling, one has obviously that 5x^/2 (^) = for t < L/2. 
Indeed, by a numerical integration of (4.1) for the short range coupling one ob- 
serves that SxL/2{t) = for times t < t* (x L; while for t > t* the perturbation, 
due to the (local) chaotic dynamics, grows as SxL/2{t) ~ Sq exp[A(i— t*)]. Thus 
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for local interactions, the predictability is mainly determined by the waiting 
time t*, necessary to have |5a;j^/2| > ^O) which is roughly proportional to the 
system size L. This is confirmed by Fig. 5, where it is shown that the average 
predictability time < Tp > as a function of L goes as 



<Tp>-- 



(4.7) 



where the time ti ~ is due to the exponential error growth after the 
waiting time and can be neglected in large enough lattices. This agrees with 
the existence of a finite speed for the perturbation spreading [212]; indeed G 
is related to the propagation velocity (see Sect. 4.6). 
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Fig. 5. Average predictability time < Tp > versus L for a CML of logistic maps 
fa{x) = ax{l — x) with a = 4 : local coupling eg = 0.3 (squares) ; non-local coupling 
(4.5) with Ci = 0.3, C2 = 0.01 a = 2 (crosses) or q = 3 (diamonds); mean field 
coupling £i = C2/L with C2 = 0.3 (crosses squares). The initial perturbation is 
applied at the center of the lattice (site z = 0) and has an amplitude 10~^^; the 
maximum admitted error is S^ax = 0.1. 

The scenario changes for non-local interactions (4.5). Now, due to the long 

range coupling, the perturbation (4.6) may propagate without any delay. The 
numerical results show that even for weak non-locality (e.g. C2 « Ci and 
rather large a- values), the waiting time t* does not increase (or increases very 
slowly) with L, so that 



< Tp >~ ti ^ X 



(4.8) 



As shown in Fig. 5, weakly non-local couplings, and mean field interactions 
[Sj = C2/N) have the same qualitative behavior. Very accurate numerical 
computations have confirmed that the dependence on L is indeed very weak 
(only logarithmic), i.e. < Tp >~ ti + alnL/Ai [213]. 



40 



This example demonstrates that the predictabihty time is given by two con- 
tributions: the waiting time t* and the characteristic time ti ~ associated 
with chaos. For non-local interactions, the waiting time practically does not 
depend on the system size L, while for local interactions it is proportional to 
L. Let us underline that in these results the non-linear terms in the evolution 
of (5x(/:) arc rather important. One numerically observes that the waiting time 
t* is not just the relaxation time Tr of 5x on the tangent eigenvector. Actually, 
Tr is much larger than t*. 

4-4 Comoving and Specific Lyapunov Exponents 

A general feature of systems evolving in space and time is that a generic 
perturbation not only grows in time but also propagates in space. Aiming 
at a quantitative description of such phenomena Deissler and Kaneko [71] 
introduced a generalization of the LE to a non stationary frame of reference: 
the comofm^' Lyapunov exponent. For the sake of simplicity, we consider again 
the case of a 1-dimensional CML. 

Let us consider an infinitesimally small perturbation initially different from 
zero only in one site of the lattice (4.6). By looking at the perturbation evo- 
lution along the hue defined by j{t) = + [vt] (where [■ ■ •] denotes the integer 
part), one expects: 

\5xj{t)\^\5xo{Q)\e^^'")^ , (4.9) 

where \{v) is the largest comoving Lyapunov exponent, i.e. 

A(^;) = lim hm lim - hi { ^^^^^ . (4.10) 

In Eq. (4.10) the order of the limits is important to avoid boundary effects. For 
V — ^ one recovers the usual LE. Moreover, one has that \{v) = A(— w) (and 
the maximum value is obtained ai v — [189]) when a privileged direction 
does not exist, otherwise X(v) can be asymmetric and the maximum can be 
attained at value v ^ (see Sect. 4.6). By writing the response function 
(2.8) in the moving frame one can also introduce the generalized comoving 
Lyapunov exponents Lg{v) for studying finite time effects [76]. 

There are other two indicators related to the comoving LE: the local Lyapunov 
exponent [183] and the specific (or temporal) Lyapunov exponents. Here we 
only briefiy discuss the latter which is indeed nothing but the Legendre trans- 
form of X{v). 
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The specific Lyapunov exponent, A(/i), has been introduced by Politi and 
Torcini [189] to characterize the growth of exponentially shaped perturbations, 
i.e. 

5x,{t) ^ Mt)e-^\'\ i^-^,...,^, (4.11) 



where ^i{t) gives the fluctuation with respect to the pure exponential shape. 
One can see that A(/i) is connected through a Legendre transform to X{v) 
[189,212]. Indeed, Eq. (4.9) defines a local exponential profile with /j, = d\{v)/dv, 
which means that in term of the specific Lyapunov exponents one expects the 
perturbation to grow according to 

5xi{t) ~ exp{A(/i)t - /ii} , i = [vt] . (4.12) 



Note that for = 0, A(/i) reduces to the standard LE. Therefore, the comoving 
Lyapunov exponent is given by 

\{v)^m-/-^- (4.13) 



The last equation defines a Legendre transform from {\{v),v) to {h.{ij),ii) 
[189]. By inverting the transformation (4.13) one obtains v — —dA{ii)/dii. 

Working in tangent space by using standard algorithms [20], one computes 
the specific Lyapunov spectrum Aj(//) with i — 1, . . . , L ior each /i. In the 
limit L — s> oo a density of such exponents exists and an interpretation of it is 
discussed in [139,140]. 



4-5 Convective chaos and spatial complexity 



So far we have considered CML's with symmetric spatial coupling, however 
there are many physical systems in which a privileged direction exists, e.g., 
boundary layers, thermal convection and wind induced water waves. The term 
usually employed for denoting such a class of systems is flow systems. See 
[8,9,39,106] for a discussion of flow systems in different physical contexts. 

In recent years it has received much attention a simplified model for fiow sys- 
tems which is able to capture the basic phenomenology [112,198,226]. A min- 
imal model is a chain of maps with unidirectional coupling [9,180,112,217,76]: 

Xn{t + 1) = (1 - c)fa{Xn{t)) + cfa{Xn-l{t)) , (4.14) 
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where t and n (= 1, . . . , L) arc the discrete time and space respectively; the 
map fa{x) is usually chosen to be the logistic map. One can consider different 
boundary conditions, XQ{t). For instance, Xo{t) = x* with x* being an unsta- 
ble fixed point of the map fa{x), or more generic time dependent boundary 
conditions where xo{t) is equal to a known function of time y{t), which can 
be periodic, quasi-periodic or chaotic. Here, following Pikovsky [180-182], we 
consider a quasi-periodic boundary condition Xo{t) = 0.5 + 0.4sin(cjt), with 
u = 7r(\/5 — 1). However, the results we are going to discuss do not depend 
too much on the details of the boundary conditions, i. e. on using XQ{t) quasi- 
periodic or chaotic. 

A central concept in the study of flow systems is the one of convective instabil- 
ity, i.e. when a perturbation grows exponentially along the flow but vanishes 
locally. 

;i(v) 

(c) 
(b) 

(a) 



Fig. 6. Sketch of the behavior of \{v) for (a) an absolutely and convectively sta- 
ble flow, (b) absolutely stable but convectively unstable flow, and (c) absolutely 
unstable flow. 

We may give a description of the phenomenology of flow systems in terms of 
the largest LE and of the comoving LE. The absolute stability is identifled by 
the condition X{v) < for all f > 0; the convective instabihty corresponds to 
Ai = X{v = 0) < and X{v) > for some velocities v > and finally standard 
chaos (absolute instability) is present when Ai = X{v = 0) > 0. See Fig. 6 for 
a sketch of the possible behaviors. 

The convective instability is conceptually very interesting, because even if the 
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Fig. 7. Evolution of a state of the system (4.14) where fa{x) is the logistic maps, 
the boundary condition is quasi-periodic, a = 3.85 and c = 0.7: in this case Ai < 
but the system is convectively unstable. 

largest LE is negative the behavior of the system may be very hard to predict, 
as Figure 7 suggests. 

For this kind of spatial "complexity" there is not a simple and systematic 
characterization. A first explanation for these features may be found in the 

sensitivity of convective unstable systems on small perturbations at the be- 
ginning of the chain (always present in physical system), which grow expo- 
nentially while they are transmitted along the flow. This simple intuition can 
be made more quantitative defining an indicator which measures the degree 
of sensitivity on the boundary conditions[217,76]. We wonder how an uncer- 
tainty |5,Xo(t)| = So in the knowledge of the boundary conditions will affect 
the system. We consider only the case of infinitesimal perturbations, i.e. 6xn 
evolves according to the tangent space dynamics, and, for the moment we do 
not consider intermittency (i.e. time fiuctuations of the comoving Lyapunov 
exponents) . 

The uncertainty 5,T„(t), on the determination of the variable at time t and site 
n, is given by the superposition of the evolved 6xo{t — r) with t = n/v: 

5xn{t) ~ j Sxo{t - T)e^^''^^dv = 5o y el^^^)/"]"^^;. (4.15) 

Since we are interested in the asymptotic spatial behavior, i.e. large n, we can 
write: 

Sxn{t) - 5oe^", (4.16) 
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The quantity T can be considered as a sort of spatial-complexity-index, an 
operative definition of which is the following: 



lim — (in -L— 







(4.17) 



where the brackets mean a time average. 

In the particular case of a non intermittent system, a simple saddle-point 
estimate of Eq. (4.15) gives 



r = max 

V 



(4.18) 



Equation (4.18) is a link between the comoving and the "spatial" Lyapunov 
exponent F, i.e. a relation between the convective instability of a system and 
its sensitivity to the boundary conditions. 

Eq. (4.18) holds exactly only in absence of intermittency; in general the rela- 
tion is more complicated. One can introduce the effective comoving Lyapunov 
exponent, ^t{v), that gives the exponential changing rate of a perturbation, in 
the frame of reference moving with velocity v, on a, finite time interval t. Ac- 
cording to general arguments (see Sect. 3.1 and [172]) one has {'yt{v)) — X{v). 
Then, instead of (4.15) one obtains 



'dv, 



(4.19) 



and therefore: 
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Therefore, because of the fiuctuations, it is not possible to write F in terms of 
X{v), although one can obtain a lower bound [217]: 



F > 
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In Fig. 8 we show F and F* vs. a for a fixed value of c. There is a large range 
of values of the parameter a for which F is rather far from F*. This difference 
is only due to intermittency, as investigations of the map fa{x) = ax mod 1 
or the computation of the generalized spatial Lyapunov exponents Ls{q) [217] 
confirm. 
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Fig. 8. r (+) and T* (Q) vs. a at a fixed value of c (c = 0.7), for the system (4.14) 
of logistic maps with quasi-periodic boundary conditions (the system is convectively 
unstable for all the considered values of the parameters). 

Concluding, we underline that the spatial complexity displayed by these sys- 
tems indicates that the unpredictability of a system cannot be completely 
reduced to the existence of at least one positive LE. 

4-6 Space-time evolution of localized perturbations 

In Fig. 9 we show the evolution of a perturbation |5xi(i)| initialized as (4.6) 
as a function of space and time {i,t), for a 1-dimensional lattice of locally 
coupled tent maps. The perturbation grows in time and propagates linearly 
in space creating a sort of predictability "horizon" : this defines a propagation 
velocity Vp [109,189,212]. 

The velocity Vp is defined in terms of the left and right edges of the disturbance 
i.e. the first left (right) site for which at time t the perturbation reaches a 
preassigned arbitrary threshold. Numerically it has been found that Vp is 
independent either of the amplitude of the initial perturbation, 5o, and of the 
threshold value, so that it is a well defined quantity [109]. 

It is easy to realize that Vp is nothing but the velocity of a moving frame of 
reference in which the perturbation is seen neither to grow nor to decrease 
(i.e. the frame comoving with the edges of the perturbation). Therefore, Vp is 
given by [109] 



The interesting point in Eq. (4.22) is that it gives not only a definite prescrip- 



X{Vp) = . 



(4.22) 
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(a) 
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Fig. 9. (a) Space-time evolution of |5xj(t)| for an initially localized perturbation 
(4.6) with do = 10-s. We used a CML of tent maps, fa{x) = a(l/2 - \x - 1/2|), 
with a = 2, e = 2/3 and L = 1001. (b) X{v) for -y > for the CML of Fig. (a). The 
straight line indicates the zero and the intersection between the curve X{v) and 
indicates the perturbation velocity Vp ~ 0.78. 

tion to derive the propagation velocity but also a link between the velocity and 
the stability properties of the system. From this point of view it is instructive 
to look at the propagation velocity in another way [212]. 

The perturbation at different times resembles a propagating front, similar 
to those encountered in reaction-diffusion processes. But while in the latter 
context the front usually separates two stable phases or a stable from an un- 
stable phase, here one phase is unstable and the other chaotic [212]. Made 
this analogy one can ask if it is possible to obtain the propagation velocity as 
in reaction-diffusion phenomena, where we know that the dynamics sponta- 
neously selects the minimum allowed velocity [128]. 

Torcini et al. [212] have shown that this is indeed the case. They studied the 
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evolution of a perturbation with an exponential profile (4.11) which, according 
to the definition of the specific Lyapunov exponent, evolves as in Eq.(4.12), 
i.e. dxi{t) ~ exp{A(/i)t — fii}. This last relation tells us that the velocity of a 
front with shape given by is V{ii) = A(yU,)//x. According to the analogy with 
reaction-diffusion systems, one expects that a generic localized perturbation 
develops an exponential decaying shape (at the edges) with a definite exponent 
fiQ (selected by the dynamics) [212]. This means that the propagation velocity 
Vp is determined by the relation Vp = V{ij,q). Now the problem is to compute 

fJ'O- 

From Eq. (4.13), which relates A(//) and X{v) through a Legendre transforma- 
tion, one obtains 



Moreover, since X{Vf) = (4.22) one has that dV/dfi = at /xq such that 
V{fio) = Vp, i.e. fiQ selects the minimal velocity. Indeed A(/i) is convex (being 
a Legendre transform), so that the minimum is unique and 



Thus for an infinitesimal perturbation, the selected velocity is the lowest pos- 
sible one [212]. 

Summarizing, for short range coupling the speed of propagation is finite and 
fully determines the spatio-temporal evolution of the perturbation. The sit- 
uation becomes different for long-range coupling as (4.5). In this case the 
velocity of propagation is unbounded [173]. For the sake of completeness, we 
mention that the long-range coupling case has been also investigated in terms 
of a specific-like Lyapunov exponent which characterizes power law shaped 
perturbations [213]. The result of this analysis shows that the perturbation 
propagates exponentially fast with a rate given by the ratio of the largest LE 
and the power of the coupling. 

We conclude this Section by mentioning that there are cases in which the 
analysis in terms of X{v) or, equivalently, A{fi) fails to give the measured 
propagation velocity. Indeed, it has been found that Vp can be larger than 
the velocity for which X{v) = 0. A finite propagation velocity has been mea- 
sured even in systems with A < (the so-called stable chaos phenomenon, see 
Section 7.3) for which the above analysis predicts Vp — [190]. 

This failure is related to the presence of strong non linearities. Recently, it has 
been proposed to generalize (4.22) to the non linear regime of the perturbation 




(4.23) 



Vp^ 




(4.24) 
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Fig. 10. max5[A((^, v)] (dashed line with points) versus v for the shift map f{x) = rx 
mod 1 with r = 1.1 and eg = 1/3, compared with X{v) (continuous hne). The two 
vertical lines indicates the velocity obtained by (4.22) which is about 0.250 and the 
directly measured one Vp ~ 0.342. Note that ma:x.s[X{d,v)] approaches zero exactly 
at Vp. 

growth by the definition of the Finite Size Comoving Lyapunov Exponent [52], 
X{d,v). It measures the divergence rate of perturbations of size 6 (not neces- 
sarily infinitesimal) in a moving frame. The algorithm is a generalization of 
the FSLE (Sect. 3.4), where now one follows an initially locafized perturbation 
along the line [vt]. In Fig. 10 we compare X{v) with X{S,v) for a CML of shift 
maps, i.e. f{x) = rx modi. The latter goes to zero exactly at the directly 
measured propagation velocity Vp. Similar results hold for other maps [52]. 
These results suggest that a generafization of Eq.(4.22), which is able to take 
into account also possible non linear effects, is: 

max [X{6, Vp)] = 0. 
s 

The numerical evidences also suggest that the condition which should be ac- 
complished in order to have deviation from the linear prediction given by 
(4.22) and (4.24) is that X{S,v = 0) > A(0,0) = A, confirming a conjecture 
done in [212]. However, even if interesting such a behavior seems to be rather 
non generic. 

4-7 Macroscopic chaos in Globally Coupled Maps 

Recently the emergence of non trivial collective behaviors in high-dimensional 

dynamical systems has gathered much attention [58, 110, 11 1,1 84]. A limit case 
of macroscopic coherence is the global synchronization of all the parts of 
the system. Beyond synchronization there exist other interesting phenomena, 
among which we just mention: clustering [11 0,129,67] and collective motion in 
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globally coupled maps (GCM) [113,202,184]. The latter behavior, in the case 
that we call macroscopic chaos [50,203] (see below), is the subject of this 
Section. 

Let us consider a globally coupled map (GCM) defined as follows 

e ^ 

xn{t + 1) = (1 - e)Uxn{t)) + - ^/„(x,(i)), (4.25) 

i=l 

where N is the total number of elements. 

The evolution of a macroscopic variable, e.g., the center of mass 

1 ^ 

m(t) ^ -^x,(t), (4.26) 

i=l 

upon varying e and a in Eq. (4.25), displays different behaviors [50]: 

(a) Standard Chaos: m{t) obeys a Gaussian statistics with a standard devi- 
ation (TAT = ,JJMff)^^^jn(^ ~ Ar-V2. 

(b) Macroscopic Periodicity: m{t) is a superposition of a periodic function 
and small fluctuations 0{N^^/'^)] 

(c) Macroscopic Chaos: m{t) displays an irregular motion as it can be seen 
by looking at the plot of m{t) vs. m{t — 1) that appears as a structured 
function (with thickness ~ N~^^^), and suggests a chaotic motion for m{t). 

Phenomena (a) and (b) also appear in CML with local coupling in high enough 
dimensional lattices [58], for the interesting case (c), as far as we know, there 
is not a direct evidence in finite dimensional CMLs. 

In the case of macroscopic chaos one can expect that the center of mass evolves 
with typical times longer than the characteristic time of the full dynamics 
(i.e. the microscopic dynamics); the order of magnitude of the latter time 
may be estimated as 1/Ai. Indeed, conceptually, macroscopic chaos for GCM 
can be thought of as the analogous of the hydro-dynamical chaos for molecular 
motion, where, in spite of a huge microscopic Lyapunov exponent (Ai ~ I/tc ~ 
lO^^s^^, Tc being the collision time), one can have rather different behaviors at 
a hydro-dynamical (coarse grained) level, i.e.: regular motion {Xhydro < 0) or 
chaotic motion (0 < Xhydro <^ Ai). In principle, if one knows the hydrodynamic 
equations, it is possible to characterize the macroscopic behavior by means of 
standard dynamical system techniques. However, in generic CML there are 
no general systematic methods to build up the macroscopic equations, apart 
from particular cases [113,184]. Therefore, here we discuss the macroscopic 
behavior of the system relying upon the full microscopic level of description. 
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The microscopic Lyapunov exponent cannot give a characterization of the 
macroscopic motion. To this purpose, recently different approaches have been 
proposed based on the evaluation of the self-consistent Perron- Frobenius (PF) 
operator [113,178,184] and on the FSLE [50,203]. Despite the conceptual in- 
terest of the former (in some sense the self-consistent PF-operator plays a 
role similar to that of the Boltzmann equation for gases [50]). here we shall 
only discuss the latter which seems to us more appropriate to address the 
predictability problem. 

We recall that for chaotic systems, in the limit of infinitesimal perturbations 
S ^ 0, one has X{S) — > Ai, i.e. X{S) displays a plateau at the value Ai for 
sufficiently small S. However, for non infinitesimal S, one can expect that 
the 5-dependence of A (5) may give information on the characteristic time- 
scales governing the system, and, hence, it could be able to characterize the 
macroscopic motion. In particular, at large scales, i.e. 6 ^ one expects 

the (fast) microscopic components to saturate and X{S) ~ Am, where Am can 
be fairly called the "macroscopic" Lyapunov exponent. 

The FSLE has been determined by looking at the evolution of \Sm{t)\, which 
has been initialized at the value (5m(i) = Smin by shifting all the elements of 

the unperturbed system by the quantity Smin (i-c. x^{0) = Xi{0) + Smin), for 
each realization. The computation has been performed by choosing the tent 
map as local map, but similar results can be obtained for other maps [203,50]. 

Figure 11a shows \{S) versus S in the case of macroscopic chaos. One has two 
plateaus: at small values of S {S < Si) , as expected from general considerations, 
\{S) = Xi, for S > S2 one has another plateau, the "macroscopic" Lyapunov 
exponent, X{S) = Am- Moreover, Si and S2 decrease at increasing A^: indeed, by 
looking at Fig. lib one can see that Si, ^2 ~ 1/y/N. It is important to observe 
that the macroscopic plateau, being almost non-existent for N = 10^, becomes 
more and more resolved and extended on large values of S^/N at increasing 
iV up to = 10^. Therefore we can argue that the macroscopic motion is 
well defined in the limit N ^ 00 and one can conjecture that in this limit the 
microscopic signature in the evolution of Sm{t) completely disappears in favor 
of the macroscopic behavior. In the case of standard chaos (Am < 0) one has 
only the microscopic plateau and then a fast decreasing of A((5)[50]. 



4.8 Predictability in presence of coherent structures 

Here we discuss some problems which arise in characterizing the predictabil- 
ity in continuous systems, described by PDE. In this case the norms are not 
equivalent [127] and the computation of the LE can give different results. 
Rather than discussing the problem in general terms, we consider here two- 
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Fig. 11. A((5) versus S for the system (4.25) with a = 1.7, e = 0.3 for iV = 10"^ (x), 
AT = 10^ (□), N = 10^ (0) and N = lO'^ (A) . The first plateaus corresponds to the 



microscopic Lyapunov exponent 



0.17 and the second one to the macro- 



scopic Lyapunov exponent Xmacro ~ 0.007. The average is over 2-10^ reaUzations for 
N = W^, 10^, 10^ and 250 reahzations for N = 10^. (b) The same as (a) rescaUng 

the 5— axis with \A/V. 



dimensional turbulence as a specific example. The choice of this example is 
due to several reasons. First of all, two-dimensional turbulence is a continu- 
ous system of relevance in atmospheric physics, and it has been extensively 
investigated in the last years [206,208,176,34]. The statistical theory for two- 
dimensional turbulence has been developed by Kraichnan and Batchelor [132] 
on a similar basis to the Kolmogorov theory for three-dimensional turbulence. 
The main formal difference is the existence of a second inviscid invariant, the 
enstrophy (average square vorticity). As a consequence, in the limit of high 
Reynolds numbers, the energy cannot be dissipated by viscosity and one ex- 
pects a direct cascade of enstrophy. With an input source at intermediate 
scales, the energy injected into the system is transferred to large scales by an 
inverse cascade. A large numbers of numerical simulations [206,34] and exper- 
iments [176] have demonstrated the universality of the inverse cascade with 
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spectral index very close to the predicted Kolmogorov exponents. 



The situation is much less clear for what concerns the direct cascade. The 
predicted spectral slope (Kraichnan-Batchelor spectrum) is seldom observed 
and even universality with respect to the forcing or to the form of dissipa- 
tion is questioned [159]. The freely decaying evolution is characterized by the 
emergence of coherent structures [159] which eventually dominate the dynam- 
ics. Coherent structures are weakly dissipative, rather regular regions of fluids 
in the turbulent background flow whose interactions can be approximately 
described by a conservative dynamics [135]. The spontaneous emergence of 
coherent structures makes two-dimensional turbulence a prototype model for 
geophysical flows [141] and, most important for our purpose, gives a natural 
example for illustrating the effects of choosing different error norms. 

The equation for describing two-dimensional turbulence is the Navier-Stokes 
equation written for the scalar vorticity a; = V x v as [18,159] 



where ip is the stream function such that v = {dyip, —dxip) and A'^ = —cu. 
As customary in direct numerical simulations, the dissipation is modified by 
employing high order viscosity p > 1 in order to achieve larger Reynolds num- 
bers. The numerical results discussed below are obtained by integrating (4.27) 
by means of a standard pseudo-spectral code on a periodic computational 
domain with resolution N x N. 

The classical theory of predictability in turbulence [136,137] studies the evo- 
lution of a difference (or error) field, defined as 

5a;(x, t) = ^ (c^'(x, t) - ^(x, t)) (4.28) 



where uj and u' are solutions of (4.27) started from slightly different initial 
conditions. The "error" is computed from Su and measured in terms of a 
given norm which is the subject of our discussion. Indeed the method used for 
defining the distance between the reference and perturbed field is a deficate 
point for continuous systems such as Navier-Stokes equations. Classical norms 
are based on the invariants of (4.27) in the inviscid limit Up — 0, i.e. enstrophy 
and energy norms [137,122] 



1 °° 
Z5{t) = - j (fx\Scu{x,t)\^ = jdkZs{k,t) (4.29) 
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Es{t) = j dkk-^ Zs{k,t). = j dkEs{k,t) (4.30) 



where we have also introduced the enstrophy (Zs) and energy (Es) error spec- 
tra. It is also natural to introduce the relative errors, defined as 



where E{t) — 1/2 / v'^{x)dx and Z — 1/2 / u;'^{x)dx, and the relative error 
spectrum 



This issue was already addressed in [122] where the infinitesimal (linear) error 
growth was computed using several "Eulerian norms" as (4.29)-(4.30). 

We will consider an initial error given by complete uncertainty at small scales, 
i.e. r{k, 0) = for A; < ko and r{k, 0) = 1 for A; > kg. This assumption is 
physically justified by the finite resolution of any measurement device and/or 
the numerical simulation scheme. For an infinitesimal perturbation, the error is 
expected to grow exponentially with the largest LE Ai. Because we are dealing 
with a dissipative system which ultimately collapses on the trivial fixed point 
a; = 0, Ai is formally negative. However, this is only a formal problem. Indeed 
in high Reynolds number turbulence the dissipation time scale is much longer 
than the dynamical time and we can make use of the effective LE 7(t) (3.8). 
For t much smaller than the dissipation time, we can consider, from any point 
of view, 7(t) as the Lyapunov exponent of the decaying turbulence. 

The exponential growth regime ends at times much smaller than the dissi- 
pative time, as soon as the separation of the two fields cannot be any more 
considered infinitesimal and the finite error regime sets in. The predictability 
time is defined by means of the accepted tolerance A or, which is equivalent, 
by a threshold for the relative errors (4.31). We will follow the classical pre- 
scription for the predictability time r{Tp) = 1/4 [137]. In Figure 12 we plot 
relative errors (4.31) as functions of time for a 512^ resolution simulation [33]. 
For small times {t < 250) we can see an exponential growth for both r{t) and 
z{t) with effective LE 7 ~ 0.08. At larger times the error curves bend and a 
predictability time estimation with energy norm gives Tp ~ 395. From Fig- 
ure 12 we learn at least two lessons. First (but not surprisingly) about half of 
the predictability time is governed by non-exponential error growth behavior. 
This is another demonstration of the little relevance of LE for characterizing 
predictability in realistic complex systems. The second observation is that the 
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Fig. 12. Relative energy (r) and cnstrophy (z) error growth for a 512^ simulation. Tp 
indicate the predictability time defined as r{Tp) = 1/4. The dashed line represents 
the exponential regime r{t) ~ exp(0.08i). 

different norms r{t) and z{t) give qualitatively similar results. Because the 
error is initially confined to small scales k > ko, the vorticity-based norm is 
always larger than the energy-based norm, but the predictability time is essen- 
tially independent of the norm used. It is not difficult to understand that any 
Eulerian norm would give comparable result. Because the error propagates 
from small to large scales, a norm which emphasizes small scale features (as 
the enstrophy norm) saturates earlier than a large scale based norm (energy, 
in our example), but the results remain essentially the same. 

In Figure 13 we plot the vorticity field of the reference a;(x, Tp) and perturbed 

field u;'(x, Tp) at the predictability time Tp. Although the two fields differ, 
by definition, by 25% in energy and about 65% in enstrophy, they look still 
remarkably similar for what concerns the distribution of vortices. Most of the 
large coherent structures are almost in the same positions. 

In Figure 13 we also plot the difference field 5c<j(x, Tp). The typical bipolar 
configuration, usually observed in simulations [122,164], indicates that the 
error is concentrated in correspondence of the vortices and that it is essentially 
due to the different position of the vortex structures in the two realizations. 

This result suggests that a Lagrangian measure of the error, based on the 
vortex positions, would be more suitable for the present system. For example, 
to emphasize the limits of the Eulerian measure for the error (4.29,4.30), 
consider the limiting case of singular point vortices, where an infinitesimal 
error in the coordinates gives error saturation and hence zero predictability 
time. In general, we expect that, in presence of vortices, an Eulerian-based 
measure underestimates the predictability time. 

This problem can be overcome by resorting to the natural distance among vor- 
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Fig. 13. Gray scale map of the vorticity fields (obtained by a 256^ simulation) at 
time Tp = 177. White corresponds to positive vorticity regions, black to negative 
ones, (a) Reference field a;(x) (b) the perturbed one uj (x) (c) the error filed 5a; (x) 

tex centers. We use a vortex tracking algorithm which recognizes and follows 
vortices during the dynamics. First we need a definition of vortex, the one here 
adopted is: a connected region Da in the computational domain with vorticity 
maximum larger (in absolute value) than a given threshold and vorticity 
larger than a fraction (we used 0.2) of the vorticity peak. Given the vortex 
domains D^, all the physical quantities are computed by integrating inside 
the domains. For example, vortex circulation is defined as = J^^ d^xa;(x) 
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and vortex center x„ is the center of mass compnted from the vorticity field. 
Finally, vortex trajectories are reconstructed by matching center positions at 
different times. A Lagrangian, vortex-based, measure of the error can, e.g., be 

10r 
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Fig. 14. Mean vortex separation d{t) at resolution 512^. At the classical predictabil- 
ity time Tp, the mean vortex separation is about one-tenth of the saturation level. 

defined as 



d\t) 



Eiral 



(4.33) 



where and x'^ are the vortex positions respectively in the reference and 
perturbed field. In Figure 14 we plot d'^ obtained from our simulation. We 

observe that at the classical predictability time, the mean vortex separation 
is d{Tp) ~ 0.5, well below the saturation value {d 

max ^ Ljl — TT in the 

periodic computational box) . This result is a quantitative confirmation of the 
observations drawn from Figure 13, i.e. the existence of an intermediate regime 
in which the (finite) error is ruled by the displacement of the strong coherent 
structures. If one is interested in predicting, with some tolerance, positions 
and intensities of coherent structures, it is possible to have a much larger 
predictability time. 



5 Predictability in fully developed turbulence 



5. 1 Basic facts of turbulence 



Perhaps, fully developed turbulence [161,84] is the most important instance of 
high-dimensional chaotic system. To give an example, let us consider a clas- 
sical experiment in fiuid dynamics: in a wind tunnel, an air mass conveyed 
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by a large fan impinges some obstacles, which perturb significantly the ve- 
locity of fluid particles. Big and small whirls appear, and the flow evolves 
irregularly in time. One could wonder whether the features of the flow depend 
crucially on the physical properties of the fluid, the size and shape of the ob- 
stacle, the mean wind velocity, and so on. It is easy to understand that, with 
a given geometry the only relevant parameter which characterizes the flow is 
the Reynolds number Re = UL/u, where U is the mean wind velocity, L is 
the typical size of the obstacle and u is the kinematic viscosity of the fluid. 
When Re is very large, i.e., of the order of a thousand or more [19] turbulence 
is called fully developed. The fundamental physical interest in this regime is 
motivated by the existence of universal properties with respect to the details 
of the experimental setup [19,161]. If a velocity probe is placed at some dis- 
tance past the obstacle, it is possible to record a temporal series that gives 
us statistical information. If one sits far enough from the obstacle, there the 
small-scale properties of the flow do not depend sensitively on the precise site 
and orientation of the probe, that is the tiirbulcnce is approximately homoge- 
neous and isotropic. Since the flow is swept across the probe at a mean velocity 
U, that largely exceeds the magnitude of the fluctuations, one can expect that 
the time record essentially amounts to a one-dimensional spatial section of the 
velocity field. Thus time-scales and length-scales are interchangeable, this is 
the essence of the Taylor hypothesis [161]. Assuming the above hypothesis, we 
can safely reinterpret temporal variations of the velocity, on an interval r, in a 
fixed-point of the space as spatial increments on scale £ = t/r, at a fixed-time. 

The first important result about the expected universality is the behavior of 
the velocity power spectrum which closely follows a power law decay E{k) oc 
^-5/3 given range of wave- numbers [123,161]. At larger wave- number the 
spectrum falls off with an exponential-like behavior, whereas the form at small 
k (i.e. large scales) depends on the mechanism of forcing and/or boundary 
conditions. For A; — > one often observes a self-similar energy spectrum E(k) ~ 
k^ with scaling exponent s > 0. In incompressible decaying turbulence, there 
are some arguments indicating that asymptotically s < 4 where the limiting 
value s = 4 is observed if initially the spectrum has s > 4 [19]. A typical 
turbulence spectrum is shown in Figure 15. 

The two crossovers unveil the presence of two characteristic scales: a large 
excitation scale L ~ k^^^, associated with the energy containing eddies, and 
a small dissipation scale in ^ k^ , related to the smallest active eddies. The 
appearance of a power law in between these two extremes unveils that no other 
characteristic scale is involved. 

A simple and elegant explanation of these experimental findings is due to 
A.N. Kolmogorov [161]: in a nutshell, it is assumed the existence of a range 
of scales where the energy - injected at the scale L - fiows down (with a 
cascade process, as remarked by Richardson [194]) to the dissipative scale 
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Fig. 15. Typical turbulent energy spectrum, /e^ is the energy containing integral 
scale and k'^ the dissipative Kolmogorov scale. 

^D1 where it is dissipated by molecular viscosity. Since, practically, neither 
injection nor dissipation takes place in this interval, it is called the inertial 
range. In this range the only relevant quantity is the average energy transfer 
rate e: dimensional counting imposes then a power spectrum E{k) oc e^l^k"^!^ 
in agreement with the experimental observations discussed above. The scaling 
for the spectrum is equivalent to a power law dependence for the second order 
structure function (SF) 



The original Kolmogorov theory (K41) assumes self-similarity of the turbulent 
flow. As a consequence, the scaling behavior of higher order structure functions 
Spipj — l\v{x + •^) — v{xyf) ~ is described by a single scaling exponent. 
The value of the exponent is determined by the so-called "4/5 law", an exact 
relation derived by Kolmogorov from the Navier-Stokes equations [123,84], 
which, under the assumption of stationarity, homogeneity and isotropy states 



where ^iv\\{Jt) is the longitudinal velocity difference between two points at dis- 
tance and e is the average rate of energy transfer. The structure function 
exponent Cp is thus predicted by Kolmogorov similarity theory to be Cp = p/3. 

Several experimental investigations [7,84] have shown that the Kolmogorov 
scaling is not exact and C,p is a nonlinear function (with (^3 = 1 as a conse- 
quence of the "4/5 law"). This means a breakdown of the self-similarity in 



(5.1) 



(5.2) 
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the turbulent cascade. Larger and larger excursions from mean values are ob- 
served as one samples smaller and smaller scales. This phenomenon goes under 
the name of intermittency [84] . A complete theoretical understanding of inter- 
mittency in Navier-Stokes turbulence is still lacking. Nevertheless, there are 
approaches, as the multifractal model [177], which are able to characterize at 
a phenomenological level the intermittency. 

In brief the basic idea of the multifractal model [177,172,84] consists in assum- 
ing a local scale-invariance for the velocity fluctuations, i.e. one has 6vi ~ i'^, 
with a continuous spectrum of (Holder) exponents h, each belonging to a given 
fractal set. In other words, in the inertial range one has 



if a; G Sh, and Sh is a fractal set with dimension D{h) and h G {h„iin, hmax)- 
The probability to observe a given scaling exponent h at the scale £ is thus 
Pe{h) ~ £3-D(/i) ^j^-g language the Kolmogorov similarity theory [123,84] 
corresponds to the case of only one singularity exponent h — 1/3 with D{h — 
1/3) = 3, see also Appendix B. 

5.2 Reduced model of turbulence 

In numerical simulations of the Navier-Stokes equations in the regime of fully 
developed turbulence, one has to discretize the original PDE to obtain a set 
of approximate ODE which must be integrated numerically. This is the direct 
numerical simulation approach which, in its simplest form, is implemented on 
a regular 3D grid of A^^ points. Since the dissipative scale (Kolmogorov scale) 
is related to the Reynolds number as in ~ LRe^^^^, an estimate of the number 
M of active spatial degrees of freedom leads to 



An obvious consequence of the fast growth of M with the Reynolds number is 
the unfeasibility of a complete turbulent simulations at high Re. The maximum 
limit of present computers is about N — 10^ which corresponds to Re ~ 10^. 

An alternative approach has been introduced with the so called shell models 
by the works of Obukhov, Gledzer and Desnyansky and Novikov (see [38] for 
a detailed discussion). The basic idea, originally motivated in the context of 
closure theory, is to implement a dynamical cascade with a set of variables 
Un {n = 1,...,N) each representing the typical magnitude of the velocity 
fluctuation in a shell of wave-numbers < |k| < kn+i. The representative 



(5.3) 



(5.4) 
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wave-numbers are spaced geometrically, /c„ = A:o2", in this way, assuming 
locality in the cascade, interactions are confined to neighboring shells only 

We will discuss a specific model, known as GOY model (see [38] for a review), 
which makes use of complex velocity variables Un and for which the equations 
of motion are 



— + Uklj Un = ikn {un+lUn+2 " -Mn-l«n+l " -Mn-2Mn-l j + fn , 

(5.5) 

where 1/ is the viscosity and /„ is a forcing term (typically restricted on the first 
shells). The coefficients in the nonlinear term (which has the same structure 
of Navier-Stokes equations) are chosen to conserve energy E = 1/2 J2n I'^nP 
in the unforced, inviscid limit. 

Without entering in the details, we recall that shell model (5.5) displays energy 
cascade d la Kolmogorov from the large scales of forcing to the dissipative 
scales (n ~ A^) with a statistical constant energy flux e. On these inertial 
range scales, the moments of velocity show power law scaling (|MnP) ~ k~'''' 
with exponents close to those experimentally observed for fully developed 
turbulence. 

The number of shells N necessary to mimic the cascade mechanism of fully 
developed turbulence is rather small, due to the geometrical progression in 
kn one has N ~ log2 Re. We have thus a chaotic dynamical system with a 
reasonably small number of degrees of freedom where standard methods of 
deterministic chaos can be used in order to relate the "turbulent" statistical 
description in terms of structure functions and intermittcncy, and dynamical 
properties, such as the spectrum of Lyapunov exponents. The absence of any 
stochastic term in (5.5) makes the shell model a natural model for investigating 
the predictability problem in turbulence. 



5.3 Effects of intermittency on predictability of infinitesimal perturbations 



The sensitive dependence on initial conditions makes the long term forecasting 
in turbulent flow practically impossible. For instance, Ruelle [196] remarked 
that thermal fluctuations in the atmosphere produces observable changes on 
a scale of centimeters after only few minutes. As a consequence after one 
or two weeks, the large scale atmospheric circulation would be completely 
unpredictable, even if the exact evolution equations were known. This is the 
so-called butterfly effect, in the words of Lorenz: A butterfly moving its wings 
in Brazil might cause the formation of a tornado over Texas. To support this 
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argument, one can observe that the largest LE of fuUy developed turbulence 
is roughly proportional to the inverse of the smallest characteristic time of the 
system, the turn-over time td of eddies of the size of the Kolmogorov length 
Id- Prom Id LRe~^/^ one obtains 

TD ~ tD/5vD ~ TL Re-^'^ , (5.6) 

where tl ^ L/U is the eddy turn-over time of the energy containing scales. As 
a consequence, as first pointed out by Ruelle [196], the largest LE scales with 
Re like A ~ I/td ~ Re^^'^/ri. Thus fully developed turbulence is characterized 
by a Lyapunov exponent diverging with Re. 

Nevertheless a large value of the LE does not prevent the possibihty of long 
term prediction, at least if one is interested in predicting the large scales 
behavior (which is related to finite errors), see Sect. 5.4. 

Remaining in the framework of infinitesimal perturbations, we discuss the 
effects of intermittency on the predictability time. The multifractal model [177] 
predicts a spectrum of viscous cut-offs: each singularity exponent h selects 
a different damping scale, ioih) ~ LRe''^/^^^^\ and hence a spectrum of 
(dissipative) turn-over times, TD{h), such that (5.6) becomes 

TD{h) ~ iD{h)/SvD ~ TLRe'^ , (5.7) 

(see Appendix B for details). To obtain the largest Lyapunov exponent now 
we have to integrate TD{h)~^, at the scale £ = £D(/i), over the /i-distribution 
Pe{h) ~ 

J T{h)-^ Pe{h)dh^ — J i-j-j dh. (5.8) 

Since the viscous cut-off vanishes in the limit it!e — > oo, the integral can be 
estimated by the saddle-point method, i.e. 

A ~ — i?e° with a = max 

Tl h 

The value of a depends on the shape of D{h). By using the function D{h) 
obtained by fitting the exponents C,q with the random /3-model [22] one finds 
a = 0.459.., slightly smaller than the Ruelle prediction a = 0.5. This result is 
confirmed by numerical simulations on the shell model (5.5) (see Fig. 16). 



D{h)-2-h 
1 + h 



(5.9) 



62 



10^ 10^ 10'' 10^ 10^ 
Re 



Fig. 16. Lyapunov exponent A (□) and variance /Lt (x) as a function of the Reynolds 
number Re for the shell model (5.5) with N = 27 shells. The dashed line is the 
multifractal prediction A ~ i?e° with a = 0.459, with function D{h) obtained by 
the random beta model fit of the (p exponents. The full line represents n ~ Re'^ 
with w = 0.8 

We remind that the fluctuations of the effective Lyapunov exponent ^{t) can 
be characterized by the ratio of n/X (Sect. 3.1). The variance n is 



where in the last expression we have introduced the integral correlation time 
tc — J C{t)dt of the effective Lyapunov exponent [66,38], where C{t) is the 
normalized correlation function of the fluctuation of 7(t) (i.e. 7(t) — A). 

The quantity ((7 — A)^) can be computed by repeating the argument for A: 



An explicit calculation [66] gives y = 1 independently of intermittency. As- 
suming that the correlation time tc vanishes as a power of Re 



I, = lim t {^{tf) - (7(t))^ ~ t.((7 - A)^) 

t— >00 L J 



(5.10) 




L 




tlRc 



—z 



(5.12) 



one ends with the prediction 



—Re 
tl 



,w 



with w — 1 — z . 



(5.13) 
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Numerical simulations on the shell model (5.5) give w ~ 0.8 (see Figure 16). 
Because w > a we obtain that fi/X diverges with Re. From Figure 16 we see 
that the strong intermittency regime begins, for the shell model, at Re ~ 10^. 
Let us stress that in the absence of intermittency one would expect that 
tc ~ and thus z — 1/2 and /i/X constant. The fact that z ~ 0.2 indi- 
cates that the presence of quiescent periods in the turbulent activity is much 
more relevant for the decay rate of time correlations than for the Lyapunov 
exponent. 

We have seen in Sect. 3.3 that the fluctuations of the effective LE affect the 
distribution of predictability time, and thus we expect a similar effect in fully 
developed turbulence. In the shell model one can estimate the predictability 
time by computing the time Tp at which the difference Sum{t) (where m cor- 
responds to the integral scale) among two realizations of the model becomes 
larger that the tolerance A. The initial difference Sq is restricted to the shell 
Uji* on the Kolmogorov scale and m <^n* . The predictability time distribution 
function is computed at two different Reynolds number. At Re = 10® we are 
at the border of the weak intermittent range: the observed PDF (Figure 17) 
is indeed close to a Gaussian with mean value 



On the contrary, at Re = 2 x 10^, the PDF exhibits the asymmetric triangular 
shape and the mean value is ruled by n according to (3.28). 

5.4 Growth of non-infinitesimal perturbations 

The classical theory of predictability in turbulence has been developed by 
Lorenz [146] (see also [143]) using physical arguments, and by Leith and 
Kraichnan [137] on the basis of closure approximations. The fundamental in- 
gredients of the Lorenz approach stem from dimensional arguments on the 
time evolution of a perturbation in an energy cascade picture. In this frame- 
work, it is rather natural to assume that the time for a perturbation at 
scale £/2to induce a complete uncertainty on the velocity field on the scale £, 
is proportional to the typical eddy turn-over time at scale i: Te ~ i/Sve where 
Svfi is the typical velocity difference at scale £. Kolmogorov scaling (5.1) gives 



Because of the geometric progression (5.15), the predictability time to propa- 
gate an uncertainty 0{Svd) from the Kolmogorov scale in up to the scale of 




(5.14) 



(5.15) 
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Fig. 17. Rescaled probability distribution functions of the predictability time Tp for 
the shell model (5.5) for (a) Re = 10^ and (b) Re = 2x 10^. The respective average 
values are (Tp) = 84.0 and 6.32 and the standard deviations are (T{Tp) = 22.2 and 
3.16. The line is the Gaussian. 

the energy containing eddies L, is dominated by the longest time 

Tp ^ + T2t^ + . . . + tl ^ tl ^ . (5.16) 



Closure approximations, where one still uses dimensional arguments, confirm 
this result [137,143]. 

It is important to stress that, in the Lorenz approach, the predictability time 
is independent of the Reynolds number. This is only in apparent contradiction 
with the increase of the Lyapunov exponent with Re (5.9). From the point of 
view of an observer interested in forecasting the large scales (i.e. not infinites- 
imal perturbations) the Lyapunov exponent is not physically relevant. Large 
scale predictability in turbulence is hence another example where a large LE 
coexists with a long predictability time. We will see that a coherent descrip- 
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tion that includes these two features of predictabihty is given by the finite size 
Lyapunov exponent (3.37). 

It is easy to estimate the scahng behavior of X{S) when the perturbation is in 
the inertial range Svd < 5 < Svl- Following the phenomenological ideas of 
Lorenz, the doubling time of an error of magnitude 6 can be identified with 
the turn-over time of an eddy with typical velocity difference 6v£ ~ 6. Using 
the scahng (5.1) one has ~ tl{(-/LY^^ ~ tl^^vi/ Svl)^"^ . In conclusion we 
obtain [11] 

\{5) - . (5.17) 

In the dissipativc range 5 < 5vd, the error can be considered infinitesimal, 
implying A (5) = A. 

Accounting for intermittency, in the framework of the multifractal approach, 
one has 

A(5) ~ J dh {6/6vLf-''^''^y^ {8/6vlY-^/^ . (5.18) 



From the basic inequality of the multifractal model D{h) < 3/i + 2 (see Ap- 
pendix B), we have 



As a result of the constancy of the energy flux in the inertial range, e = v'^{£) / £, 
the equality holds for h = h*{3), and gives 3h*{3)+3 — D{h*{3)) = 1. Therefore 
a saddle point estimation of (5.18) gives again (5.17). The dimensional scaling 
of the FSLE in fully developed turbulence X{S) ~ 5~'^ is thus not affected by 
intermittency corrections. This is a direct consequence of the exact result (5.2) 

These findings have been numerically tested on the shell model (5.5) for the 
energy cascade. Figure 18 shows the scaling of {1/T{6v,r))t as a function of 
5v in the GOY model, where T{Sv,r) is the "doubling time", i.e. the time 
necessary for a perturbation of size Sv to increase by a factor r (see Sect. 3.4 
and Appendix A). 

For comparison we also plot the eddy turn-over times rf^ — {\5v£\'^)^^'^ / £. We 
see that below the Kolmogorov scale, the doubling time displays a constant 
plateau corresponding to the Lyapunov exponent (3.39). At larger errors we 
observe a good agreement with the prediction (5.17). Let us observe that, even 
at this high Reynolds number, the scaling range for the doubling time is rather 
small. 
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§u 

Fig. 18. The inverse of the error doubling times versus 5u (diamond) compared 
with shell turn-over times (plus). Number N of simulated shells is 27, and Reynolds 
number Re = = 10^, /cq = 0.05. The initial perturbation is randomly uniform 
over all shells in the inertial range, with amplitude of order 10"^. The first threshold 
is do = 10~^ and the error growth rate parameter r is 2^/^. The number of error 
doubling experiments is 400. The dashed line has the slope —2. 

It is interesting to look at the doubling time as a function of the Reynolds 
number. For small thresholds the inverse of the doubling time scales as the 
Lyapunov exponent, i.e. roughly as Re~^/'^. We also observe that the bend 
away from the infinitesimal growth rate occurs at smaller scales for larger 
Reynolds numbers. This suggests the following scaling ansatz: times and errors 
are scaled with the turn-over time and the typical scale of fluctuations at the 
Kolmogorov scale, that is by Re"^''^ and Re~^^^, respectively. In Figure 19 we 
show the re-scaled data. The data collapse is reasonable, allowing to conclude 
that small-scale predictability, with small error amplitudes, behaves (apart 
from intermittency corrections) as predicted by Ruelle [196], whereas large- 
scale predictability, characterized by large error amplitudes, is well described 
by Lorenz arguments. 

To improve the data collapse, taking into account the multifractal correction 
as described in Appendix B, one has to make a multiscaling collapse, i.e. to 
rescale ln(l/T(5v, r)) and hi{5v/VQ) with ln(it!e/it!eo) where Vq and Rcq are 
two parameters to be fixed [12]. The result is shown in Figure 20. The data 
collapse is clearly improved. 

Finite size predictability has been investigated also in two-dimensional turbu- 
lence, which is relevant for atmospheric flows. As discussed in Sect. 4.8, two- 
dimensional turbulence in the inverse energy cascade regime is characterized 
by a scaling a la Kolmogorov [132] with no intermittency [34]. As discussed 
above, the scaling exponent in (5.17) is not affected by intermittency; however 
intermittency does reduce the scaling range because of the intermediate dis- 
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Fig. 19. In [(l/r((5ti,r))/ReV2j versus In [(5n/Re-^/^" 

Re = (O) N = 24 and Re = 10^; (+) N = 27 and Re = 10^; (□) = 32 and 
Re = 10^°; (x) iV = 35 and i2e = 10^^ Tfie straight line has slope -2. 
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Fig. 20. ln(l/r(5'U,r))/ln(Re/i?o) versus ln(5'u/V^)/ ln(Re/i?o); multiscaling data 
collapse at different Reynolds numbers Re = v^^. The fitting parameters are 
= 6 X 10^ Vo = 5x 10-2, and Re = w'^. 

sipative range (see Appendix B). The absence of intcrmittcncy corrections in 
2D turbulence suggests that the dimensional scaling (5.17) is observable even 
in direct numerical simulations at moderate Reynolds number. 

Let us consider two realizations of the vorticity field in (4.27) starting from 
very close initial conditions. The error S is defined, following (4.30), as 6{t) = 
\J Esif). In Figure 21 it is shown the FSLE \{5). It is remarkable the rather 
wide scaling range for \{5) ~ 5^^ with respect to the shell model simulations 
(Fig. 18) obtained at much larger Re. As a consequence of the absence of 
intermittency, also the crossover from the infinitesimal regime A (5) = A to the 
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Fig. 21. Finite size Lyapunov exponent X{6) as a function of velocity uncertainty 6 in 
a direct numerical simulations with 1024^ grid points of 2D turbulence in the inverse 
cascade regime. The asymptotic constant value for 5 ^ is the largest Lyapunov 
exponent of the turbulent flow. The dashed line has slope —2. In the inset we show 

the compensated plot A((5)(5^/e. 

inertial range regime (5.17) is sharp. 

Prom a general point of view, it is interesting to observe that even in the 
absence of intermittency, fixed scale analysis based on the FSLE overpasses 
fixed time analysis in the characterization of predictability. Dimensional con- 
siderations and closure approximations [137] predicts a linear growth of the 
error in the inverse energy cascade as 

Es{t)^Gst, (5.20) 



where G is an adimensional constant. It is easy to realize that (5.20) is equiva- 
lent to (5.17), X{6) having the dimension of an inverse time and 6 = y/Eg. The 
result obtained in numerical simulations is shown in Figure 22, which has to be 
compared with Figure 21. The scahng law (5.20) in Figure 22 is barely visible, 
making the determination of G difficult. On the contrary, inverting (5.17) to 
(5.20) one can measure G directly from Figure 21. The result obtained is in 
close agreement with closure computations [37]. 



5.5 e-entropy for turbulent flows 



A complementary way to look at the predictability of turbulent flows is in 
terms of its entropy (see Sects. 2.1.2 and 3.5). 

Unfortunately a direct measurement of the Kolmogorov- Sinai entropy is prac- 
tically infeasible. Indeed for i?e — > oo due to the huge number of active degrees 
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Fig. 22. Average energy error {Es{t)) growth. Dashed hne represents hnear closure 
prediction, dotted hne is the saturation value E. The initial exponential growth is 
emphasized by the lin-log plot in the inset. 

of freedom, the KS-entropy diverges, so that one needs velocity measurements 
with an extremely high resolution and lasting for extremely long times, far be- 
yond the actual experimental possibilities. Nevertheless, limiting the analysis 
to not very high resolution, one can hope to extract some interesting piece of 
information by investigating the behavior of the e-entropy, h{e). As far as the 
e-entropy of turbulence is concerned, two questions can be raised. 

i) Since a direct measurement of the full 3-dimensional velocity field is infea- 
sible, one has usually access just to a time signal measured in one spatial 
point: which kind of information can we extract from the e-entropy per unit 
time of such a signal? 

ii) Taking into account i), can we say something about the e-entropy of the 
full 3-dimensional velocity field? 

In ii) we are referring to the e-entropy, /i'^^(e), per unit time and volume (the 
symbol ^'^ means space-time). In other words, we are assuming that the total 

entropy of a turbulent fiow observed for a (very long) time T on a (very large) 
volume V of the 3-dimensional space has the form H{V,T,e) ~ VTh^^{e). 
See Ref. [89] for an introduction of this concept. 

Both in i) and ii), as we will see, a crucial role is played by the sweeping of 
the large scales of the fiow on the small ones, i.e. the Taylor hypothesis (see 
Sect. 5.1). 

5.5.1 e-entropy for a time signal of turbulence 

In order to estimate the e-entropy of a given signal one has to compute the 
Shannon entropy of the symbolic sequence obtained by making an (e, r) grid 
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in phase-space (Sect. 3.5). Unfortunately, this method is rather inefficient for 
signals in which many scales are excited [3,4,51], e.g., as in turbulence. There- 
fore, here we resort to a recently proposed method [3] based on the exit-time 
analysis. 

In a few words, the idea consists in looking at a sequence of data not at 
fixed sampling times but at fixed fiuctuation (see Appendix C), i.e. when the 
fluctuation of the signal exceeds a given threshold, e. In practice we code the 
signal v{t) of total duration T in a symbolic sequence fl'^ {e) = {ti(e), ki{e)}f£^, 
where tj(e) is the first times such that \v{tQ + YX^i'^ki^) + — '^{to + 

Z]fc=i^fc(e))| > e/2 (being to a reference time) and ki = ±1 tells us in which 
direction (up or down with respect to f (to + Z]fe=i^fc(e))) the fiuctuation has 
been realized. M is the total number of exit events, i.e. Z)^iti(e) = T. Note 
that Jl^(e) is a faithful coding of the signal within the required accuracy e. 
Now the evaluation of the entropy goes as usual through the evaluation of the 
Shannon entropy, h^{e), of the sequence Jl^(e). Finally the e-entropy per unit 
time is given by [3]: 

, , , h^ie, Tr) 

where a coarse-graining of the possible values assumed by t{e) with a res- 
olution time Tr has been considered, and (t(e)) is the average exit time, i.e. 
(t(e)) = (1/M) Y.i=i^M ti{^)- The formula (5.22) is exact in the limit — > (in 
Appendix C one finds the derivation of (5.21) and the details of the method). 

This procedure allows a noticeable improvement of the computational possi- 
bility to measure the e-entropy. In particular, if one is interested in the leading 
scahng behavior of /i(e) with e, one only needs to estimate the scahng of (t(e)). 
Indeed, the correction induced by h^{e, Tr) can be shown to be sub-leading (in 
particular, logarithmic). 

Now, we estimate the average exit time for the velocity signal v{t). This can 
be done assuming the Taylor hypothesis and the multifractal model (see Ap- 
pendix B). In this framework we can assume that, for t corresponding to 
scales R = Ut in the inertial range, the following relation holds \()tv\ = 
\v{to + t)— v{t)\ ~ t^ and each h is picked with probability P{h) ~ i^~D{h)_ 
Since we are interested in the statistics of the first times necessary to observe 
a fiuctuation \5tv\ ~ e, one can "invert" the above relation [30]: 

t{e) ~ e^/^ with P{h) ~ e(3-^('^»/'^ . (5.22) 
The exit-time moments [30], also called inverse structure functions [107], can 
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be estimated in the multifractal framework as follows 



{{t'ie))) ~ J dhe"^^^ ~ e^^i^ (5.23) 



where x(g) may be obtained with a saddle point estimate in the limit of small 
e: 



X{q) = min 



h 



q + 3-D{h) 
h 



(5.24) 



The average ([. . .]), obtained by counting the number of exit-time events M, 
and the average (([...])), computed with the uniform time sampling are con- 
nected by the relation 



where the term ti/J2jLitj takes into account the non- uniformity of the exit- 
time statistics. Therefore the quantity we arc looking for, i.e. the mean exit- 
time, is given by {t{e)) — {{t~^{e)))~^ ~ (e)~^*^~^^. By noting that 

-l + ^'-«<"'-2-^("'>-3torall/., (5.26) 



h h 

which is nothing but Eq. (5.19), i.e. the 4/5 law of turbulence, we finally obtain 

h{e) ~ e"^ . (5.27) 

In Fig. (23) we report the evaluation of the upper and lower bound (see Ap- 
pendix C) of h{e) for a synthetic signal, v{t), constructed in such a way to 
reproduce the statistical properties of turbulence [29] . 

Let us now compare the above results with a previous study of the e-entropy 
in turbulence [224], where it was argued that: 

h{e) ~ , (5.28) 



a behavior that differs from the prediction (5.27). The behavior (5.28) has 

been obtained by assuming that h{e), at scale e, is proportional to the inverse 
of the typical eddy turnover time at that scale: since the typical eddy turnover 
time for velocity fluctuations of order ~ e is r(e) ~ e^, Eq. (5.28) follows. 
Indeed this is the argument used to derive (5.17) for the FSLE. The difference 
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Fig. 23. Numerically computed lower bound (□) and upper bound (o), with 
r = 0.1(t(e)) for the (e, r)-entropy in the case of a multiafHne signal with ^(3) = 1. 
The signal has been obtained with the method of Ref. [29] (see also Appendix D) us- 
ing a D{h) which fits experimental data at large Reynolds number. The two straight 
lines show the theoretical scaling e~^. 

between (5.28) and (5.27) can be understood by considering that even if X{5) 
and h{e) are two complementary concepts (the fact that for both the estimate 
of the scaling behavior reduces to the "4/5 law" is not a coincidence), in the 
latter case one has to consider the sweeping induced by the large scales. On 
the contrary, since the former is related to the distance of two realizations 
which differ in the small scales (< 6) but not on the large scales {> S), the 
sweeping of the large scales is not effective. 



5.5.2 e-entropy of turbulence and the Taylor Hypothesis 

Now we study the e-entropy per unit time and volume for the velocity field of 

turbulent flows in 3 + 1 dimensions, h'^'^{e). We will show that, by assuming 
the usually accepted Taylor hypothesis, one has a spatial correlation which 
can be quantitatively characterized by an "entropy" dimension T> = 8/3. As 
already remarked, h^'^{e) cannot be directly measured so we will discuss its 
estimation in a theoretical framework by introducing a multi-affine field. For 
the sake of simplicity, we neglect intcrmittency by assuming a pure self-affine 
field with a unique Holder exponent h = 1/3. 

Let us first introduce a multi-affine field with the proper spatial and tem- 
poral scaling [4]. The idea consists in defining the signal as a dyadic three- 
dimensional superposition of wavelet-like functions (/^((x — x„^k(^))/^n) whose 
centers move due to the sweeping. The coefficients of the decomposition a„,k(^) 
are stochastic functions chosen with suitable self-affine scaling properties both 
in time and in space. A field with spatial Holder exponent h in d-dimensions 
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is (see Appendix D) : 

M 2''("-i) / _ 



vM^Y. E ar.,,{t)A M , (5.29) 



n=l fe=l \ 



where x„ ^ is the center of the k wavelets at the level n. i.e. for eddies with 
size In ~ 2"". According to the Richardson-Kolmogorov cascade picture, one 
assumes that sweeping is present, i.e., x^+i ^ = x^^/ + v^+i^k where {n,k') 
labels the "mother" of the (n + 1, /c)-eddy and r^+i ^ is a stochastic vector 
which depends on r„ fc/ and evolves with characteristic time t„ oc (£n)^~'*. 

If the coefficients {a„,fc} and {r„,fc} have characteristic time r„ ~ (^n)^"'^ a-nd 
{on.fe} ~ (^n)'*, it is possible to show (see Appendix D for details) that the 
field (5.29) has the correct spatio-temporal statistics, i.e. 



|^;(x + R,io) -'y(x,io)|~|R|'' (5.30) 
\v{K,to + t) -v{K,to)\r^t'' . (5.31) 

In addition the proper Lagrangian sweeping is satisfied. Now we are ready for 
the e-entropy analysis of the field (5.29). If one wants to look at the field v 
with a resolution e, one has to take n in (5.29) up to N given by: 

(iN)'' - e , (5.32) 



in this way one is sure to consider velocity fluctuations of order e. Then the 
number of terms contributing to (5.29) is 

#(e) ~ (2^^)^ ~ . (5.33) 



By using a result of Shannon [201] one estimates the e-entropy of the single 
process a„_jt(i) (and also of r„j) as: 




(5.34) 



where the above relation is rigorous if the processes a„,fc(^) are Gaussian and 
with a power spectrum different from zero on a band of frequency ~ 1/t„. 
The terms which give the main contribution are those with n ^ N with 
Tn ~ {iNy~^ ~ e^~h-\ Their number is given by (5.33) so that, collecting the 
above results, one finds 

h'T^e) r^i^r. . (5.35) 

Tn 
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For the physical case d — 3, h — 1/3, one obtains 



(5.36) 



By denoting with v^j the typical velocity at the Kolmogorov scale 77, one has 
that Eq. (5.36) holds in the incrtial range, i.e., e > ~ Re~^^'^, while for 
e < Vjj, h^^{e) = constant ~ i?e^^/'^. 

Let us now consider an alternative way to compute the e-entropy of the field 
v(x,t): divide the rf-volume in boxes of edge length £(e) e^/^ and look at 
the signals f (xo,,t), where the x^^ are the centers of the boxes. Denoting with 
/i*^") (e) the e-entropy of the temporal sequence of the velocity field measured 
in Xc^, we have 

^e-iA (5.37) 

because of the scaling (5.31). Therefore, h^'^{e) is obtained summing up all 
the "independent" contributions (5.37), i.e. 

h^^{e) ~ AA(e)/i(") (e) ~ M{e)e-'/'' , (5.38) 

where A/'(e) is the number of independent cells. It is easy to understand that 
the simplest assumption J\f{e) ~ l{eY ~ e'^/'* gives a wrong result, indeed one 
obtains 

h^^ie) ~ e-"^ , (5.39) 

which is not in agreement with (5.35). In order to obtain the correct result 
(5.36) it is necessary to assume 

M{e) - l{ef , (5.40) 

with V = d — h. In other words, one has that the sweeping implies a non- 
trivial spatial correlation, quantitatively measured by the exponent V, which 
can be considered as a sort of "entropy" dimension. Incidentally, we note that 
V has the same numerical value of the fractal dimensions of the velocity iso- 
surfaces [154,218]. From this observation, at first glance, one could conclude 
that the above result is somehow trivial since it is simply related to a geomet- 
rical fact. However, a closer inspection reveals that this is not true. Indeed, 
one can construct a self-affine field with spatial scaling h and thus with the 
fractal dimension of the velocity iso-surfaces given by d — h for geometrical 
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reasons, while V — d. Such a process can be simply obtained by eliminating 
the sweeping, i.e., 

^(x,i) = E E an..W^(^^^) , (5.41) 

n=l k=l V -f-n / 

where now the ^n,k are fixed and no longer time-dependent, while an,k ^ i^n)'^ 
but r„ ~ £„. 

We conclude by noting that it is possible to obtain (see [89]) the scaling (5.35) 
using equation (5.41), i.e. ignoring the sweeping, assuming Tn ~ {^nY''^ and 
ctn,fc ~ i^n)'^', this corresponds to take separately the proper temporal and 
spatial spectra. However, this is not satisfactory since one has not the proper 
scaling in one fixed point (see Eq. (5.37) the only way to obtain this is through 
the sweeping). 



6 Uncertainty in the evolution equations 

The study of a large class of problems in science (physics, chemistry, biol- 
ogy,...) is reduced to the investigation of evolution laws, which describe some 
aspects of the system. The assumption that natural processes can be described 
by mathematical models is at the foundation of this approach [220,70]. The 
purpose of this Section is to discuss how the unavoidable uncertainty in the 
equation of motion puts limits on the long time forecasting. 

To be more concrete, let us consider a system described by a differential equa- 
tion: 

|x(i)=f(x,i), x,fe]R". (6.1) 

As a matter of fact, we do not know exactly the equations, so we have to devise 
a model which is different from the true dynamics. In practice this means that 
we study 

4- x(t) = fe(x, t) where fe(x, t) = f (x, t) + e5f (x, t) . (6.2) 
at 

Therefore, it is natural to wonder about the relation between the true evolu- 
tion {reference or true trajectory xy(t)) given by (6.1) and the one effectively 
computed {perturbed oi motie/ trajectory XM(t)) given by (6.2). A typical ex- 
ample is the relation between the true dynamics of the physical system and 
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the one obtained by a computer simulation. This issue is of particular rele- 
vance for the study of weather forecast where it is referred to as predictability 
of the second kind [175]. 

In this context it is particularly relevant the shadowing lemma [41] which 
implies that, for Anosov systems, a computer may not calculate the true orbit 
but what it does find is nevertheless an approximation of the true one. As 
a consequence, the statistical properties are well reproduced by an accurate 
numerical integration [96]. 



A central point in the discussion of the second kind predictability problem 
is the issue of structural stability [99]: since the evolution laws are known 
only with finite precision it is highly desirable that at least certain properties 
were not too sensitive to the details of the equations of motion. For example, 
in a system with a strange attractor, small generic changes in the evolution 
laws should not change drastically the statistical properties of the dynamics 
[74,105]. 



In order to see that a non generic perturbation, although very "small" in 
some sense, can produce dramatic changes in the statistical properties of the 
dynamics, following Refs. [28,105], we consider the one-dimensional chaotic 
map x{t -|- 1) = f{x{t)) with f{x) = ix mod 1, and a perturbed version of it: 



8x 



8x-f 



X e 



X e 



5 247 
8' 384 

247 265 
384' 384 



(6.3) 



265 17 
384' 24 



X e 

4a; mod 1 otherwise . 



The perturbed map is identical to the original outside the interval [5/8, 17/24], 
and the perturbation is small in L2 norm. Nevertheless, the fixed point x = 
2/3, which is unstable in the original dynamics, becomes stable in the per- 
turbed one, and it is a global attractor for fe{x), i.e. almost every point in 
[0, 1] asymptotically approaches x — 2/3. 

If one compares the trajectories obtained iterating f{x) or fe{x) it is not 
difficult to understand that they may remain identical for a certain time but 
unavoidably differ utterly in the long time behavior. The transient chaotic 
behavior of the perturbed orbits can be rendered arbitrarily long by reducing 
the interval in which the two dynamics differ [28]. 

As for the problem of predictability with respect to perturbations on the 
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Fig. 24. The map fe of equation (6.3) (solid line) and the original chaotic map / 
(dashed line). 

initial conditions, the problem of second kind predictability in the limit of 
infinitesimal perturbations is essentially understood in terms of the Lyapunov 
exponents. Indeed, it is possible to show (see below) that a small uncertainty 
on the evolution laws of chaotic systems has the same effects of an error of 
the same order of magnitude on the initial conditions. However, also in the 
case of second kind predictability one has often to deal with errors which are 
far from being infinitesimal. Moreover, in real systems the size of an uncer- 
tainty on the evolution equations is determinable only a posteriori, based on 
the ability of the model to reproduce some of the features of the phenomenon. 
Typical examples are systems described by partial differential equations (e.g. 
turbulence, atmospheric flows). The numerical study of these systems is per- 
formed by using a model with unavoidable severe approximations, the most 
relevant due to the necessity to cut some degrees of freedom off (i.e. the small 
scale variables) . A relevant problem in this case is to quantify the effect of the 
unresolved scales on the predictability of the resolved ones. 

Prom a general point of view, in the second kind predictability problem we can 
distinguish three main cases depending on the original dynamics. In particular, 
Eq. (6.1) may display: 

(i) trivial attractors: asymptotically stable fixed points or attracting periodic 
orbits; 

(ii) marginally stable fixed points or periodic/quasi-periodic orbits as in inte- 
grable Hamiltonian systems; 

(iii) chaotic behavior. 

In case (i) small changes in the equations of motion do not modify the qualita- 
tive features of the dynamics. Case (ii) is not generic and the outcome strongly 
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depends on the specific perturbation Si, i.e. it is not structurally stable (see 
[64] for a discussion on this point). In the chaotic case (iii) one expects that 
the perturbed dynamics is still chaotic. In the following we will consider only 
this latter case. 



In chaotic systems, the effects of a small uncertainty on the evolution law is, for 
many aspects, similar to those due to imperfect knowledge of initial conditions. 
As an example let us consider the Lorenz system (3.29). In order to mimic 
an indetermination in the evolution law we assume a small error e on the 
parameter r: r ^ r + e. Let us consider the difference (5x(t) = XM(t) — ^rit), 
for simplicity, (5x(0) = 0, i.e. we assume a perfect knowledge of the initial 
condition. For small e one has, with obvious notation: 



dSx 



fe(xM) - f(xT) 



^ 5x + — e 
9x dr 



(6.4) 



Since at time t = one has |5x(0)| = 0, |5x(t)| initially grows under the 
effect of the second term in (6.4). At later times, when |5x(t)| fti 0(e) the first 
term becomes the leading one, and we recover the first kind predictability 
for an initial uncertainty 5q ~ e. Therefore, apart from an initial growth, 
which depends on the specific perturbation, for small enough e the evolution of 
(ln(|5x(t)|)) follows the usual linear growth with the slope given by the largest 
LE. Typically the value of the LE computed by using the model dynamics 
differs from the true one by a small amount of order e, i.e. Am = Ay + 0(e) 
[64]. 




0.0001 



Fig. 25. Finite Size Lyapunov Exponents XrTiS) (+) and Atm('^) (x) versus S for 
the Lorenz model (3.29) with a = c = 10, b = 8/3, r = 45 and e = 0.001. The 
dashed line represents the largest Lyapunov exponent for the unperturbed system 
(At ~ 1.2). The statistics is over 10^ realizations. 

A picture of the error growth, valid also for finite errors, can be obtained 
by considering the Finite Size Lyapunov Exponent. In addition to the FSLE 
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of the model, Xmm{S), we introduce the FSLE for the true dynamics (6.1) 
Xtt{S) (which cannot be measured in real situations) and XTAii^), the FSLE 
computed following the distance between one true trajectory and one model 
trajectory starting at the same point. In the case of a perfect model Xmm{S) ~ 
Xtt{S). The results of the computation for the Lorenz model (3.29) are shown 
in Figure 25. Xtt{S) displays the chaotic plateau with A ~ 1.2. As discussed 
above, for S > e the second term in (6.4) becomes negligible and wc observe 
Xtm{S) — Xtt{S) ~ A. In this range of errors the model system recovers 
the intrinsic predictabihty of the true system. For very small errors, Xj^m is 
dominated by the second term in (6.4) and deviates from A^r- 



6.1 Uncertainty introduced by numerical computations 

In numerical computations, an unavoidable source of errors is due to the rep- 
resentation of numbers on the computer, as computers work with integers. 
This has two main consequences: the phase space of the simulated system 
is necessarily discrete (and finite); and the computation introduces a sort of 
noise due to the round-off. 

A direct consequence of the discreteness in phase space is that any numerical 
trajectory is periodic. At first sight, this seems a very serious problem, es- 
pecially when integrating chaotic systems which have non periodic behavior. 
However, as discussed in [64], apart from cases in which one uses very low pre- 
cision, and very low dimensional systems, the period is usually extremely large 
and one works with an effective continuous phase space dynamical system (see 
Sect. 7.1). 

The round-off produces on (6.1)-(6.2) a perturbation (5f (x, t) of order e ~ 10~" 
{a ^number of digits in floating point representation) which depends on f and 
on the software [131]. In general, the round-off error is very small and may 
have a positive role in selecting the physical probability measure, the so-called 
natural measure, from the set of the admissible invariant ones [74] . 

In order to show the effect of the numerical precision on the predictability, 
let us consider again the Lorenz model (3.29). At variance with the previous 
Section, here we assume to have a perfect knowledge of the model (i.e. of the 
parameter r), and the error is introduced only by the numerical integration, 
e.g. by different time step At. The most precise integration with smallest At 
is taken as the reference trajectory and the other is the perturbed one. The 
result is shown in Figure 26 for two different values of At for the perturbed 
integration. In both cases, for small values of the error, the exponential growth 
rate is given by the largest LE A. The same behavior is observed by introducing 
the numerical error in other ways, e.g. by using different precision (single or 
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Fig. 26. (ln|5x(i))| versus t, where |(5x(i)| is the Euchdean distance between two 
trajectories of the Lorenz model (3.29) for r = 28. Curve (a) refers to the comparison 
between trajectories obtained using a fourth order Runge-Kutta algorithm with 
At = 4 X 10~'^ and At = 4 x 10~^. Curve (b) shows the same quantity obtained 
with At = 4 X 10""^ and At = 4 x 10"^. The dotted line with slope A « 0.9 is shown 
for comparison. 

double) or different integration algorithms [64]. 

6.2 Finite resolution effects and parameterization of unresolved scales 

Let us now consider more complex situations, in which many interacting de- 
grees of freedom and different characteristic times are involved [36]. We will 
consider the particular examples of an extremely simplified model of global 
circulation [147,148] and the shell model (Sect. 5.2). 

For systems with many different scales usually one is able to represent only 
the large scale variables. A typical situation is the discretization of partial dif- 
ferential equations. The small scale modes, below the computational grid, are 
unresolved and are typically parameterized according to some phenomenolog- 
ical prescription (e.g. the eddy viscosity parameterization of the small scales 
[141,84]). So we consider systems of the following form 



where x e H" represent the large (and typically slow) variables while y e IR"* 

represent the small (and fast) ones. As explained above, in many practical 
situations the small variables cannot be explicitly resolved. In this framework, 
a natural question is: how must we parameterize the unresolved modes in order 
to predict the resolved ones? In this respect, the optimal parameterization is 



dx. 
'di 



f(x,y) 



dy 
dt 



g(x, y) 



(6.5) 
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that one for which the predictabihty on the resolved modes is not worse than 
the intrinsic predictability of the same variables in the complete system, i.e. 
in our notation Atm = )^tt- 

An example in which it is relatively simple to develop a model for the small 
scale modes is represented by skew systems, i.e., g depends only on the fast 
variables y. In this case, simply neglecting the fast variables or parameter- 
izing them with a suitable stochastic process does not drastically affect the 
prediction of the slow variables [32] . 

On the other hand, in typical cases y feels some feedback from x, and, there- 
fore, we cannot simply neglect the unresolved modes (see Ref. [36] for details). 
In practice one has to construct an effective equation for the resolved variables: 

^ = fM(x,y(x)), (6.6) 



where the functional form of y(x) and is built by phenomenological argu- 
ments and/or by numerical studies of the full dynamics (if available). 

Let us now discuss a simplified model for atmosphere circulation [147,148] 
which includes large scales (synoptic scales) and small scales Uj^k (convective 
scales) : 



-Xk-l {Xk-2 - Xk+l) -UXk + F - ELi Vj, 



k 



dt - - V ^ '^-r-. - ■ ^r- ^^^^^ 



dyj,k 

dt 



-cbyj+i^k {yj+2,k - yj-i,k) - cvyj^k + Xk , 



where k = 1, ...,K and j = 1, J. As in [147] we assume periodic boundary 
conditions on k {xx+k = Xk, yj,K+k = yj,k) while for j we impose yj+j,k = 
yj^k+i- The variables Xk represent some large scale atmospheric quantities in 
K sectors extending on a latitude circle, while the yj^k represent quantities on 
smaller scales in J ■ K sectors. The parameter c is the ratio between fast and 
slow characteristic times and b measures the relative amplitude (both larger 
than unity). Model (6.7), even if rather crude, contains some non trivial aspects 
of the general circulation problem, namely the coupling among variables with 
very different characteristic times. 

Being interested in forecasting the large scale behavior of the atmosphere by 
using only the slow variables, a natural choice for the model equations is: 

^ = -Xk-i (xk-2 - Xk+i) -uxk + F - Gfe(x) , (6.8) 



where Gk{x.) represents the parameterization of the fast components in (6.7). 
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Fig. 27. Finite Size Lyapunov Exponents for the Lorenz model (6.7) Xtt{S) (solid 
line) and Xtm{S) versus S obtained by dropping the fast modes (+) and with eddy 
viscosity parameterization (x). The parameters arc F = 10, K = 36 , J = 10, u = 1 
and c = b = 10, implying that the typical y variable is 10 times faster and smaller 
than the x variable. The value of the parameter f e = 4 is chosen after a numerical 
integration of the complete equations as discussed in Ref . [36] . The statistics is over 
10^ realizations. 

Following the approach discussed in Ref. [36], a physical reasonable parame- 
terization is 



where is a numerically determined parameter. 

In Figure 27 we plot Xtm{S) obtained from different choices of Gk- The sim- 
plest possibility is to neglect the fast variable, i.e. Gk = 0. Also for large 
errors we have Xtm{S) > Xtt{S) because this crude approximation is not able 
to capture the intrinsic predictability of the original system. More refined pa- 
rameterizations in terms of stochastic processes with the correct probability 
distribution function and correlation time do not improve the forecasting abil- 
ity. On the contrary Eq. (6.9) gives the result shown in Figure 27. At small 
scales we still observe deviations from A^t but, at variance with the previous 
case, we recover intrinsic predictability for error of the size of the resolved 
scale. 

As a more complex example, let us consider a version of the shell model 
discussed in Sect. 5.2, more precisely we study [151]: 




(6.9) 



dm 



'n 




dt 



(6.10) 
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with n — 1, . . . ,N. 
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Fig. 28. The FSLE for the eddy- viscosity sheU model (6.11-6.12) Amm(5) at various 
resolutions Nm = 9(+); 15(x), 20(*). For comparison it is drawn XttI^) (continuous 
line). Here k = 0.4, ko = 0.05. 



At variance with the previous example, here we have a set of scales in — ^/kn 
with characteristic times t„ ~ (see Sect. 5.4). In order to simulate a 

finite resolution in the model, we consider a model of (6.10) in terms of an 
eddy viscosity [24]: 



du. 



dt 



-l^jC^klUn + fn, (6.11) 



where now n = 1, ...,Nm < N and the eddy viscosity, restricted to the last 
two shells, has the form 



where k is a constant of order 1 [24]. In the model equation Nm < N the 
molecular viscosity term is much smaller than the eddy viscosity term and can 
be simply neglected. Model equations (6.11-6.12) are essentially the large eddy 
simulation for the shell model. Thus, although shell models are not realistic 
models for large scale geophysical flows (being nevertheless a good model for 
small scale turbulent fluctuations), the study of the effect of truncation in 
term of eddy viscosity is of general interest. 

In Figure 28 we show Xmm{S), i.e. the FSLE computed for the model equations 
(6.11) with = 24 at different resolutions Nm — 9,15,20. A plateau is 
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Fig. 29. The FSLE between the eddy-viscosity sheh model and the fuU shell model 
Xtm{S), at various resolutions Nm = 9(+), 15(x), 20(*). For comparison it is drawn 
\tt{S) (continuous line). The total number of shell for the complete model is 
N = 24, with ko = 0.05, u = 10"^. 

detected for small amplitudes of the error 5, corresponding to the LE, which 

2 /3 

increases with the resolution according to A ~ ^nIj- At larger 5, the curves 
collapse onto the Att(^), showing that large-scale statistics of the model is 
not affected by small-scales resolution. 

The ability of the model to predict satisfactorily the features of the "true" dy- 
namics is not anyway determined by \mm{^) but by \tm{5)^ which is shown 
in Figure 29. Increasing the resolution Nm = 9, 15, 20 towards the fully re- 
solved case N — 24 the model improves, in agreement with the expectation 
that Xtm approaches Xtt for a perfect model. At large 5 the curves practically 
coincide, showing that the predictability time for large error sizes (associated 
with large scales) is independent of the details of small-scale modeling. 

6.3 Lyapunov exponents and complexity in dynamical systems with noise 

We saw how in deterministic dynamical systems there exist well established 
ways to define the complexity of a temporal evolution, either in terms of the 
Lyapunov exponents and the Kolmogorov-Sinai entropy, or by means of their 
generalization to non infinitesimal perturbations, like FSLE and e-entropy. 
The situation is much more ambiguous with random perturbations, which are 
always present in physical systems as a consequence of thermal fiuctuations or 
hidden changes of control parameters, and, in numerical experiments, because 
of the roundoff errors [162]. 

The combined effect of the noise and the deterministic part of the evolution 
law can produce highly non-trivial behaviors [43,59,73,102,103,157]. Let us 
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mention stochastic resonance, where there is a synchronization of the jumps 
between two stable points [25,26,163] (for a recent review see [87]), and the 
phenomena of the so called noise-induced order [157] and noise-induced insta- 
bility [43,59]. 

When facing systems with noise, the simplest possibility is to treat the random 

term as a time-dependent term, that is to consider the separation of two close 
trajectories with the same realization of noise. In this way one computes the 
largest LE, A^-, associated with the separation rate of two nearby trajectories 
with the same realization of the stochastic term (where o indicates the noise 
strength). Although A^- is a well defined quantity, i.e. the Oseledec theorem 
[169] holds, it is not the most useful characterization of complexity. In addition, 
a moment of reflection shows that it is practically impossible to extract A^- 
from experimental data. 

We will show how, for noisy and random systems, a more natural indicator 
of complexity can be obtained by computing the separation rate of nearby 
trajectories evolving with different noise realizations. This measure of com- 
plexity, defined in [174,149], and inspired by the contributions of Shannon 
[201] and Kolmogorov [126], is related to the mean number of bits per unit 
time necessary to specify the sequence generated by a random evolution law. 



6.3.1 The naive approach: noise treated as a standard function of time 

The approach in which one treats the random term as an usual time- dependent 
external force can lead to misleading results, as illustrated in the following 
example. 

Let us consider a one-dimensional Langevin equation 

dx dV(x) f— ,^ 



where r\{t) is a white noise and V{x) diverges for | a; |— > oo, like, e.g., the 
usual double well potential V — — x^/2 -|- x^/4. 

The Lyapunov exponent Ao-, associated with the separation rate of two nearby 
trajectories with the same realization of 7^(i), is defined as 

A^ = lim -ln|^(i)| (6.14) 

t— >oo t 
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where the evolution of the tangent vector is given by: 

dz d^V{x{t)) 



Since the system is ergo die with invariant probabihty distribution P{x) ~ 
Q^-v{x)ic^ one has: 

A, = hmt^^ \ In \z{t)\ = - hmt_oo \ \l dl,V{x{t'))dt' = 

(6.16) 

-C Jdl^V{x)e-^^''y^ dx = -f /(a^V^(a;))2e-^(^)/^ dx<Q. 



This result has a rather intuitive meaning: the trajectory x{t) spends most of 
the time in one of the "valleys" where —d1^V{x) < and only short intervals 
on the "hills" where —dl^V{x) > 0, so that the distance between two trajec- 
tories evolving with the same noise realization decreases on average. Notice 
that in Rcf. [215], supported by a wrong argument, an opposite conclusion has 
been claimed. 

A negative value of implies a fully predictable process only if the realization 
of the noise is known. In the case of two initially close trajectories evolving 
under two different noise realizations, after a certain time T^-, the two trajec- 
tories can be very distant, because they can be in two different valleys. For 
(7 — >■ 0, due to the Kramers formula [57], one has T^- ~ exp AV/a, where AV is 
the difference between the values of V on the top of the hill and at the bottom 
of the valley. The result obtained for the one dimensional Langevin equation 
can easily be generalized to any dimension for gradient systems if the noise is 
small enough [149]. 

Another example showing the limitations of this approach is provided by the 
case of stochastic resonance in chaotic systems. In this case, in fact, one can 
find the same qualitative behavior both for a positive and a negative LE. We 
refer to [149] for more details. 



6.3.2 An information theory approach 

The main difficulties in defining the notion of "complexity" of an evolution law 
with a random perturbation already appears in ID maps. The generalization 
to A'"-dimensional maps or to ordinary differential equations is straightforward. 

Therefore, we consider the model 

x{t + l) = f[x{t),t]+aw{t), (6.17) 
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where t is an integer and w{t) is an uncorrclatcd random process, e.g. w are 
independent random variables uniformly distributed in [—1/2,1/2]. For the 
largest LE A^-, as defined in (6.14), now one has to study the equation 

z{t + l)^f'[x{t),t]z{t), (6.18) 

where f — df/dx. 

Following the approach of Sect. 2.3, let x{t) be the trajectory starting at 
x{0) and x'{t) be the trajectory starting from x'{Q) = x{Q) + 5x{Q). Let Sq = 
|5x(0)| and indicate by ti the minimum time such that \x'{ti) — x{ti)\ > A. 
Then, we put x'{ti) = x{ti) + Sx{0) and define T2 as the time such that 
\x'{ti + T2) — x{ti + T2)| > A for the first time, and so on. In this way the 
Lyapunov exponent can be defined as 

A = = In (^) (6.19) 
T \doJ 

being t = Yl,T~i (sec also Appendix A). If the above procedure is applied by 
considering the same noise realization for both trajectories, A in (6.19) coin- 
cides with (if A<j > 0). Differently, by considering two different realizations 
of the noise for the two trajectories, we have a new quantity 

if„=il„(^), (6.20) 

which naturally arises in the framework of information theory [5] and algo- 
rithmic complexity theory. The times ti,T2, . . . are nothing but the intervals 
at which it is necessary to repeat the transmission of x(t), with a precision 5o, 
and Kfj/ In 2 is the number of bits per unit time one has to specify in order to 
transmit the sequence. If the fiuctuations of the effective Lyapunov exponent 
7(t) are very small (i.e. weak intermittency) one has: 

= A + 0((7/A). (6.21) 

The interesting situation happens for strong intermittency when there are 
alternations of positive and negative 7 during long time intervals: this induces 
a dramatic change for the value of K^r- This becomes particularly clear when we 
consider the limiting case of positive 7^^^ in an interval Ti >> 1/7^''^^ followed 
by a negative 7^^^ in an interval T2 >> 1/17^^)1 , and again a positive effective 
LE and so on. During the intervals with positive effective LE the transmission 
has to be repeated rather often with ~ Ti/(7*^-'^^ In 2) bits at each time, while 
during the ones with negative effective LE no information has to be sent. 
Nevertheless, at the end of the contracting intervals one has \5x\ = 0{a), 
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so that, at variance with the noiseless case, it is impossible to use them to 
compensate the expanding ones. This implies that in the limit of very large 
Tj only the expanding intervals contribute to the evolution of the error 6x{t) 
and is given by an average of the positive effective Lyapunov exponents: 

X,~(7^(7)). (6.22) 



Note that it may happen that K^^ > with < 0. We stress again that (6.22) 
holds only for strong intermittency, while for uniformly expanding systems or 
rapid alternations of contracting and expanding behaviors K^^ ~ X^. 

Note that K^r is a sort of e-entropy (see Sect. 3.5), indeed, the complexity we 
consider is defined for Sq not too small {5o ^ cr) . If 6o and A are small enough, 
but still much larger than a, K„ is essentially independent of their values. 

The relation K„ ~ il^^il)) is the time analogous of the Pesin relation (2.15) 
hxs < J2i The latter relation expresses the fact that negative Lya- 

punov exponents do not decrease the value of hxs, because the contraction 
along the corresponding directions cannot be observed for any finite space par- 
tition. In the same way the contracting time intervals, if long enough, do not 
decrease K^. Another important remark is that in the usual treatment of the 
experimental data, where noise is usually present, one practically computes 
and the result can be completely different from Ao-. 
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Fig. 30. versus T with a = 10~^ for the map (6.23). The parameters of the 
map are a = 2 and 6 = 2/3 (squares) or b = 1/4 (circles). The dashed lines are the 
noiseless limit of K^. 

Let us now briefly discuss some numerical results obtained with two differ- 
ent systems (Fig. 30 and Fig. 31). The first example consists in a periodic 
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Fig. 31. Act (squares) and K„ (crosses) versus a for the map (6.24). 
alternation of two piecewise linear maps of the interval [0, 1] into itself: 



ax modi if (2n - 1)T < i < 2nT; 
hx mod 1 if 2nT <t<{2n+ 1)T 



(6.23) 



where a > 1 and 6 < 1. Note that in the limit of small T, — > max[Ao-, 0], 
because it is a non-negative quantity as shown in Fig. 30. 

The second example (Fig. 31), strongly intermittent without external forc- 
ing, is the Beluzov-Zhabotinsky map [103,157], introduced for describing the 
famous chemical reaction: 



{1/8-xy/^ + a\e-'= + b if < a; < 1/8 



X 



- 1/8)1/3 + al e-^ + 6 if 1/8 < a; < 3/10 (6.24) 



,c(10xe-io^/3)i9 + 6 



if 3/10 < X 



with a = 0.50607357, b = 0.0232885279, c = 0.121205692. The map exhibits a 
chaotic alternation of expanding and contracting time intervals. In Figure 31, 
one sees that while X„ passes from negative to positive values at decreasing a, 
K„ is not sensitive to this transition [157]. Considering the system with a given 
reahzation of noise as the "true" evolution law, one has that \a corresponds 
to Xtt while K„ corresponds to Xtm- 

The previous results show that the same system can be regarded either as 
regular (i.e. A^- < 0), when the same noise realization is considered for two 
nearby trajectories, or as chaotic (i.e. K^r > 0), when two different noise 
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realizations are considered. 



6.4 Random dynamical systems 



We discuss now dynamical systems where the randomness is not simply given 
by an additive noise. This kind of systems has been the subject of interest in 
the last few years in relation to the problems involving disorder [119], such as 
the characterization of the so-called on-off intermittency [187] and to model 
transport problems in turbulent flows [227,228,86]. In these systems, in gen- 
eral, the random part represents an ensemble of hidden variables believed to 
be implicated in the dynamics. Random maps exhibit very interesting features 
ranging from stable or quasi-stable behaviors, to chaotic behaviors and inter- 
mittency. In particular on-off intermittency is an aperiodic switching between 
static, or laminar, behavior and chaotic bursts of oscillation. It can be gen- 
erated by systems having an unstable invariant manifold, within which it is 
possible to find a suitable attractor (i.e. a fixed point). For further details we 
refer to [187]. 

A random map can be defined in the following way. Denoting with x(t) the 
state of the system at discrete time t, the evolution law is given by 

x(i + l) = f(x(i),J(i)), (6.25) 



where J{t) is a random variable. 



As for the case of additive noise examined in the previous Section, the sim- 
plest approach is the introduction of the LE Aj computed considering the 
separation of two nearby trajectories evolving with the same realization of the 
random process J{t) = ii,?2i ■■■,it- The Lyapunov exponent \j generalizes A^- 
of Sect. 6.3.1 and can be computed from the tangent vector evolution: 

A,= lim ^ln|z(Ar)| (6.26) 

Af— >oo iV 



where 

Zm{t + 1) = E ^^^^^§^^^n(^) . (6.27) 



On the other hand, also for these systems, as in the case of additive noise, it 
is possible to introduce a measure of complexity, K, which better accounts for 
their chaotic properties [174,149] 

Kc^hsh + Xj0{Xj), (6.28) 
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where hsh is the Shannon entropy of the random sequence J{t). The meaning 
of K is rather clear: K/ In 2 is the mean number of bits, for each iteration, nec- 
essary to specify the sequence Xi,...,Xt with a certain tolerance A. Note that 
there are two different contributions to the complexity: (a) one has to specify 
the sequence J(l), J(2), J{t) which implies hsh/^'^'^ bits per iteration; (b) 
if A,y is positive, one has to specify the initial condition a;(0) with a precision 
Aexp^^"'"^, where T is the time length of the evolution. This requires A,// In 2 
bits per iteration; if \j is negative the initial condition can be specified using 
a number of bits independent of T. 



6.4-1 A toy model: one dimensional random maps 

Let us discuss a random map which, in spite of its simplicity, captures some 
basic features of this kind of systems [187,101]: 

x{t + l) = atx{t){l-x{t)), (6.29) 



where at is a random dichotomous variable given by 

_ J 4 with probability p , . 

~ 1 1/2 with probability 1-p. ^ ' 



For x{t) close to zero, we can neglect the non linear term to obtain 

t-i 

x{t) = Y[ ajx{0) ; (6.31) 

j=0 



from the law of large numbers one has that the typical behavior is 

x{t) ~ x(0)e<i°">*. (6.32) 



Since (In a) = pln4+(l— p) In 1/2 = (3p— 1) In 2 one has that, forp < pc = 1/3, 
x{t) — > for i — > oo. On the contrary for p > Pc after a certain time x{t) 
escapes from the fixed point zero and the non-linear term becomes relevant. 
Figure 32 shows a typical on-off intermittency behavior for p slightly larger 
than Pc- Note that, in spite of this irregular behavior, numerical computations 
show that the LE Aj is negative for p < pc — 0.5: this is essentially due to the 
non-linear terms. 

By introducing a finite threshold e, in order to discriminate laminar and in- 
termittent phases, we can define a complexity K{e). We denote with II and 
Ij the average life times respectively of the laminar and of the intermittent 
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Fig. 32. x{t) versus t for the random map (6.29 - 6.30), with p = 0.35. 

phases for p close to Pc « II)- The mean number of bits, per iteration, 
one has to specify in order to transmit the sequence is [150] 



Ijh 



J'I'Sh 



h hsh 



K{e) ^ ^ 

ln2 {lj + lL)ln2 iiln2' 



(6.33) 



To obtain (6.33) first notice that on an interval T one has approximatively 
T/{lj + II) intermittent bursts and the same number of laminar phases. Then 
notice that, during a laminar phase, there is not an exponential growth of 
the distance between two trajectories initially close and computed with the 
same sequence of Oj. Since during a laminar phase one has to send a number 
of bits which does not depend on its duration, one can send all the neces- 
sary information simply by giving the sequence of at during the intermittent 
bursts. Eq.(6.33) has an intuitive interpretation: in systems with a sort of 
"catastrophic" events, the most important feature is the mean time between 
two subsequent events. 



6.4-2 Sandpile models as random maps 

Another example of a system which can be treated in the framework of random 
maps is represented by the so-called sandpile models [15]. These models are a 
paradigmatic example of the Self-Organized Criticality (SOC) [14]. This term 
refers to the tendency of some large dynamical systems to evolve spontaneously 
toward a critical state characterized by spatial and temporal self-similarity. 
The original Sandpile Models are probabilistic cellular automata inspired to 
the dynamics of avalanches in a pile of sand. Dropping sand slowly, grain 
by grain on a limited base, one reaches a situation where the pile is critical, 
i.e. it has a critical slope. This means that a further addition of sand will 



93 



produce sliding of sand (avalanches) that can be small or cover the entire size 
of the system. In this case the critical state is characterized by scale-invariant 
distributions for the size and the lifetime and it is reached without tuning of 
any critical parameter. 

We will refer in particular to the Zhang model [229] , a continuous version of the 
original sandpile model [15], defined on a rf-dimensional lattice. The variable 
on each site Xi (interpretable as energy, sand, heat, mechanical stress etc.) can 
vary continuously in the range [0, 1] with the threshold fixed to Xc — 1. The 
dynamics is the following: 

(a) one chooses at random a site and adds to it an energy 5e, 

(b) if at a certain time t the energy in a site, say i, exceeds the threshold Xc 
a relaxation process is triggered defined as: 

Xi ^ , 

where nn indicates the 2d nearest neighbors of the site i; 

(c) one repeats point (b) until all the sites are relaxed; 

(d) one goes back to point (a). 

Let us now discuss the problem of predictability in sandpile models on the 
basis of the rigorous results [46], which clarify the role of the LE for this class 
of systems. 

In Ref. [46] it has been proved that the LE Xj is negative. In fact the dynamics 
of a fittle difference between two configurations follows the same rules (a)-(d), 
i.e., the "error" is redistributed to the nearest neighbors site, so that one has 

A. < (6.35) 



where R is the diameter of the system. 

As for other examples already discussed, the existence of a negative LE does 
not mean a perfect predictability. This can be understood by looking at the 
growth of the distance, S{t), between two initially close trajectories computed 
with two different realizations of randomness, i.e., by adding sand in different 
sites. Let us consider the case of the "minimal error" : in the reference realiza- 
tion one adds sand on a site i chosen at random. In the perturbed realization, 
instead, one adds a sand grain at one of the nearest sites of i. In such 
S{t) increases up to a maximal distance in few avalanches [150]. Practically, 
one has the same kind of phenomenon, already discussed, of the Langevin 
equation with two noise realizations. 
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Let us now estimate the complexity K of this system. An upper bound can 
be given by using (6.28) K = hgh + ^jdi^j), where hsh is the entropy of the 
random sequence of addition of energy. In sandpile models, since each site 
has the same probability to be selected, one has hsh — InF, where V is the 
number of sites of the system. Since the Lyapunov exponent is negative, the 
complexity is just determined by hsh- 



7 Irreguleir behavior in discrete dynamical systems 

For the sake of completeness we include in this review a discussion on the 
characterization of irregular behaviors in systems whose states are discrete. 
Such systems include Cellular Automata (CA), which have been intensively 
studied both for their intrinsic interest [223] and for applications as, e.g., to 
simulate hydrodynamic equations [82] or to study various forms of chemical 
turbulence [166,167,40]. Other interesting systems with discrete states are the 
neural networks used for modeling some brain functions [6]. It is also relevant 
to note that in every simulation with a computer, because of the finite number 
of digits it can use, one deals with a system with discrete states (see Sect. 6.1). 
In addition, the general problem of dynamics of systems with discrete states is 
important in the debated issue of quantum chaos. Indeed quantum mechanics 
can be regarded as a discretized version of the classical one, acting on a suitable 
lattice in phase space, where the number of the possible states is proportional 
to the inverse of the Planck constant [81,49,60]. 

If a system consists of N elements and each element can assume an integer 
number k of distinct values, = is the number of states. When these 
states evolve with a deterministic rule, the dynamics can be depicted in terms 
of oriented graphs: a set of points, representing the states, are connected by 
arrows, indicating the time evolution. Of course, each point has one, and only 
one, outgoing arrow; but different arrows can end at the same point. For any 
finite system each initial condition evolves to a definite attractor, which can 
be either a fixed point (as in Fig. 33a), or a periodic orbit (Fig. 33b). 

In systems of this kind, obviously, it is not possible to use the previously in- 
troduced indicators of chaos, e.g. the Lyapunov exponents or the Kolmogorov- 
Sinai entropy, whose definitions rely on the continuous character of the system 
states. Moreover, the asymptotic periodic behavior seems to force the conclu- 
sion that discrete states systems are trivial, from an entropic or algorithmic 
complexity point of view. 

The above conclusions, although mathematically correct, are rather unsatis- 
factory from the physical point of view, indeed from this side the following 
questions deserve some interest: 



95 



• ♦ 





s 




o 




Fig. 33. Schematic representation of the evolution of a deterministic rule with a 
finite number of states: (a) with a fixed point , (b) with a periodic cycle. 

(1) What is the "typical" period, T, in a system with N elements, each 
assuming k distinct values ? 

(2) When T is very large, how can we characterize the (possible) irregular 
behavior of the trajectories, on times that are large enough but still much 
smaller than T ? 

(3) What does it happen in the transition from discrete to continuous states, 
i.e. in the limit k ^ oo 1 

In the next subsections we will deal to the above questions. 

7.1 Dependence of the period on the number of the states 

For deterministic discrete state systems the dependence of the period of the 
attractor on the number of the states, may be addressed with a statistical ap- 
proach in terms of random maps [63] . We recall that this problem is important 
for computer simulations of chaotic systems (see Sect. 6.1). li M — k^ ^ 1 is 
the number of states of the system, the basic result for the average period, T, 
is 



In the following we give a simple argument, by Coste and Henon [63]. 

For simplicity of notation, we consider the case with /c = 2, so that the state 
of the system is a string of bits. A deterministic evolution of such a system 
is given by a map which is one among the possible functions connecting the 
2^ states. Let us now assume that all the possible functions can be extracted 
with the same probability. Denoting with I{t) the state of the system, for a 
certain map we have a periodic attractor of period m if I{p + m) = I{p) and 
Hp + i) 7^ Hp) J i < The probability, a;(m), of this periodic orbit is 
obtained by specifying that the (p+m — 1)*'* first successive images of the map 




(7.1) 
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are distinct from all the previous ones and the {p + mY^ iterates coincides with 
the p*'* one. Since one has I{p + 1) 7^ I{p + m), with probability (1 — l/A/"); 

I{p + 2) 7^ I{p + m), with probability (1 - 2/J\f); ; J(p + m - 1) 7^ 

I{p + m), with probability (1 — (m — 1)/J\f); and, finally, /(p + m) = /(p) 
with probability (l/A/"), one obtains 



The average number, M(m), of cycles of period m is 

M(m) = — u;(m) ^ ft ^ ^ , (7.3) 

m m 



from which one obtains T ~ vCV" for the average period. 

It is here appropriate to comment on the relevance of Eq. (7.1) for computer- 
generated orbits of chaotic dynamical systems. Because of the finite number, 
n, of digits used in the floating point representation, when one iterates a 
dynamical system, one basically deals with a discrete system with a flnite 
number M of states. If d2 indicates the correlation dimension of the system 
[93,94], one can reasonably assume that M ~ lO"''^^, so that, from Eq. (7.1) 
one has: 



This estimation gives an upper limit for the typical number of meaningful 
iterations of a map on a computer. Note that this number, apart from the 
cases of one or two dimensional maps with few digits, is very large for almost 
all practical purposes. 



7.2 The transition from discrete to continuous states 



Following the basic ideas of Ford [79,80], as discussed in Sect. 2.3, and the 
results of Sect. 6 - on the predictability in systems whose evolution law is not 
completely known - we describe now a way to introduce a practical deflnition 
of chaos for systems with discrete states. In addition, we deal with the problem 
of the transition from discrete to continuous states. 

Given a system with N possible states, denoting with l{t) its state at time t 
we can write its evolution law as: 

I(t + l)=F[I(t)]. (7.5) 
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A single state I is a sequence of (at most) ln2 A/" bits, and its time evolution for 
M steps can be surely translated in a binary sequence E of length i^{M, J\f) < 
MlnaAA. 

Relying one the definition of algorithmic complexity (Sect. 2.2.3) we can make 
the following classification: we call regular (compressible) those sequences that 
can be encoded by a computer program whose length ^^(M, A/") increases 
less than linearly in M, when M ranges over a physically significant inter- 
val, at fixed values of A/". Otherwise the system will be called chaotic or in- 
compressible. Let us call ip the binary length of the algorithm for one step: 
£f < 2A/'ln2A/'. The sequence E can be expressed by the record composed by 
the initial state 1(0) (specified by ln2 Ambits), the number of steps M (specified 
by ln2M bits) and the rule F for one step (specified hy ip bits). Therefore 

ej:{M,J\f) < (2A^+l)ln2A^ + ln2M + 0(l). (7.6) 

Let us note that from the above equation one has that - when M grows 
indefinitely and J\f is constant - is logarithmically bounded and hence the 
sequence appears to be compressible. This is somewhat trivial since, because of 
the discrete nature of the states, the motion at M > A/" (in practice M > T ~ 
VAT) is periodic. Therefore it is interesting to consider only 1 -C M < T ^ A/". 
Although the evolution law (7.5) can be carried out, in principle, in exact 
arithmetic, in practice in real computations one has unavoidable errors due 
to truncations and approximations. Let us now regard the evolution law (7.5) 
as a computer program with input 1(0) and a set C of parameters, with C 
components, needed to define the algorithm F. If these parameters are all 
known within precision 0(2~^), the binary length of the coding of C is 0{qC). 

Consider the following problem: given two identical initial conditions I*^^^(0) = 
I*^^)(0) = 1(0), and two different realizations C*^^^ and C*^^-* of the set of coeffi- 
cients C (with difference 0(2~^)), what is the dependence on e = 2~^ of the 
first-error time M (i.e. the first time for which ^^^\t) ^ l^'^\t))l Of course, 
the answer depends on the realizations of the components of C and on the 
initial conditions 1(0). Let us consider C*^^^ as an e-perturbation of C^^-*, i.e. 
we pose, for each component of the parameter vector: 

= Cf ) + e, , (7.7) 

where the random variables are uniformly distributed in [— 2(~*~^), 2^"^"^)]. 
Let us note that the coding length 0{Cq) + \n2M is enough to define the 
sequence E up to the first error time M . Performing an average on the 
and on the initial conditions 1(0), one can compute an average first-error time 
(M(e)), and a typical first-error time Mtyp{e) = exp(lnM(e)). For (M) and 
Mtyp the dependence on e can be of the following type: 
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(a) (M) ~ e"^ ~ , 

(b) (M) ~ln2(l/e) ~ g. 



In the case (a) we say that the system is compressible, while if (6) holds one 
has a chaotic (incompressible) case. The above classification is rather obvious: 
in case (a) a trajectory of length M can be coded by a program of length 
©(InaA/") +0(lnM), while in case {h) one has a length 0(ln2 A/") + 0(M). For 
a detailed discussion see Ref. [65]. 

Let us now discuss in an explicit example the previous approach and the prob- 
lems in the transition to the continuous state case. We consider a discretized 
standard map, as obtained by considering lattice points in the torus [0, 27r]^ of 
the form (a;, y) with x = 2'kQ/ L and y = 2tiP/L, where P and Q are integers 
between 1 and L. The action of the map is 

Q{t + 1) = [ Q{t) + a ^ sin(P(i) f ) ] mod L 

(7.8) 

P{t + 1) = {P{t) + Q{t + l)) modL 

where a is the control parameter and [•] means integer part. Prom the results 
of Sect. 7.1 one has that the typical period for the map (7.8) is 71 ~ L; so 
if L is large the periodic motion will be seen only for sufficiently large times. 
In the system (7.8) one has just one parameter, i.e. the "kick strength" a. 
Numerical evidence supports the following picture: at fixed L, the first-error 
time is roughly constant for large values of the error e, while it goes as 
for small errors e. The transition between the two regimes occurs at a critical 
value ec(I/) which scales as ec ~ l/L. In formulae: 

It is rather easy to give analytical estimates supporting the numerical evidence 
[65] 

^ f (^) for e < / 

where the angle 9 is defined via sin^^ = V^j ^'^d / = 2ti/L. Numerical simula- 
tions show that the behavior proposed in Eq. (7.9) holds. To have a comparison 
with the usual standard map, we have computed the average time {M/\{e)) re- 
quired for two trajectories to reach lattice points farther than a fixed distance 
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A in the discrete phase space of Eq. (7.8). We found: 

(MA(e))~|Sy^)^"^^/^)^^>^^Sf| (7.11) 
^ ^ " [1/e for e < ec(L). ^ ' 

We remark that when e < ec(-L), M/^{e) is weakly dependent, i.e. logarithmi- 
cally, on A. This is just another verification of the similarity of the effect of 
a small disturbance on the equations of motion and of a small error in the 
initial conditions for a dynamical evolution (see Sect. 6). 

These results unveil the nature of the dynamics of this discrete system: its 
trajectories are incompressible and therefore chaotic only for large values of e, 
the cutoff value decreasing as 1/ L. This helps us also to understand the extent 
to which the dynamics of the discrete standard map Eq. (7.8) is equivalent 
to its continuum counterpart. When a is large, and e > ec, the two systems 
possess chaotic trajectories. Simple calculations show that, to the leading or- 
der, M ~ log2(-L). After this time, the discrete system appears "regular", i.e. 
compressible and predictable. Therefore, continuous and discrete systems are 
similar (as far as chaos is concerned) only over logarithmically short times. It 
is important to stress that the system appears "regular" on time scales much 
smaller than the typical period 71 ~ ^^^2/2 (^i^gj^g correlation dimension 

of the attractor [93,94]). 

Recently Mantica [155] studied the algorithmic complexity in classical polyg- 
onal billiards with L sides. The system, for any finite value of L, is regular; on 
the other hand, as L — > 00, it tends to a curved billiard, which can be chaotic. 
This system is very similar to the discrete dynamical system (7.8) and may 
be used to study the transition from quantum to classical mechanics and the 
principle of correspondence. The average complexity of symbolic trajectories 
in the polygonal billiards has the same scaling behavior (as function of L and 
of the precision e) of that one of the system (7.8), i.e. a compressible (regular) 
regime for e < ec ~ 1/L and an incompressible (chaotic) one for e < Ce- 
ll is interesting to note that a similar feature is characteristic of quantum 
dynamics of systems whose classical behavior is chaotic. Roughly speaking, 
a quantum system behaves as a system with discrete states whose number is 
proportional to ^~ . A semi-classical wave function follows a dynamics which 
is approximately classical up to a time tc ~ (1/A) ln(///i), where A is the 
Lyapunov exponent of the classical motion, and / is a typical action of the 
motion. Over this time, the quantum system has the same complexity of its 
classical counterpart, while for larger times its quantal (quasi-periodic) nature 
appears [81,49,60]. 
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1.3 Entropies and Lyapunov exponents in Cellular Automata 

Cellular automata (CA) consist of discrete valued state variables, ai{t), defined 
on a discrete lattice, and updated synchronously at discrete time according to 
a local rule. They can be defined in any dimensions and for any finite number 
of possible values for o'j(t). For the sake of simplicity we consider Boolean CA, 
i.e. (Ti(t) = {0, 1}, in a 1-dimensional lattice. An evolution rule can be written 
as: 

ai{t + l) ^F[ai^r{t),...,ai(t),...,ai+r(t)], i^l,...,N, (7.12) 

where r defines the range of the coupling, i.e. the variable in a site depends on 
the variables in the 2r neighbor sites. If F in (7.12) only depends on the sum of 
the state variables, one speaks of "totalistic" CA. Another usual requirement 
is to have symmetric rules. For further details we refer to Ref. [223], where 
the standard scheme for the classification of the possible rules and a number 
of examples of CA-behavior are discussed. 

In the following we refer to Id Boolean cellular automata with local symmetric 
rules - as those ones systematically studied by Wolfram [223] . 

7.3.1 Classification of Cellular Automata according to the transient times 

For finite lattices with N sites the number of possible states of CA is finite 
and equal to J\f — 2^. As already discussed, this means that, strictly speaking, 
from an entropic (or algorithmic) point of view CA arc trivial. Therefore the 
problem of the characterization of irregular behaviors in CA has, in principle, 
some meaning only in the limit N ^ oo. In more physical terms, for finite 
one expects the characterization in terms of entropy to be possible for times 
shorter than the typical period T{N) or the typical transient time T{N), 
provided T{N) and T{N) are long enough. 

Actually, cellular automata behaviors can be classified according to the de- 
pendence of T(N) and T{N) on A^". One has three possible classes of behavior. 

Regular cellular automata (class 1 and 2 in Wolfram's classification [223]) 
evolve either on homogeneous states both in time and space (the analogous 
of fixed point in dynamical systems) either to a set of simple stable periodic 
structures (analogous to limit cycles) which, in general, depend on the initial 
configuration (Fig. 34a). In these CA, T{N) and T{N) can range from being 
almost independent of to be, at maximum, proportional to N . 

C/iao^zc cellular automata (class 3 in [223]) yield disordered patterns (Fig. 34b). 
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Fig. 34. Typical behavior of (a) regular [Rule 52], (b) chaotic [Rule 22], (c) complex 
[Rule 20]. We used totalistic r = 2 cellular automata. Time flows from below to 
above 

For any finite N these CA reach periodic states, but there are rather clear nu- 
merical evidences that the transient time T increases exponentially with the 
system size: 

f(N) ~ exp(cA^). (7.13) 

Moreover, also the cycle period shows in most of the cases a similar dependence 
on N, this is a reminiscence of what we discussed in Sect. 7.1. 

Complex cellular automata (class 4 in [223] , Fig. 34c) usually evolve toward 
complex localized structures (gliders) which interact in a complicate way. For 
these CA numerical simulations [100] have shown that both the transient time 
and the cycle period display a non trivial A^-dependence (i.e. the average, the 
typical values or the median depend in a different way on A^). The unpre- 
dictability of these system manifests itself in the distribution of these times. 
In particular, the large variability of these times in dependence of the initial 
conditions and the lattice size inhibits any forecasting of the duration of the 
transient. 

In the following we limit the discussion to chaotic rules, i.e. class 3 in the Wol- 
fram classification. A detailed characterization of complex CA would require 

the introduction of concepts and tools that are beyond the aim of this review, 
for further details see Refs. [13,96,100,223]. 

7.3.2 Sensitive dependence on initial conditions 

A first intuitively reasonable characterization of irregular behaviors is in terms 
of sensitive dependence on initial conditions, but in CA it is not possible to 
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have arbitrary small distances between two states. Nevertheless, for large A^, 
when considering two states with only one different element, one can say that, 
in some sense (i.e. in an appropriate norm), the difference is small. Denoting 
with Rt the number of different elements at time t, we can define the damage 
propagation speed as [223] 

R 

v^lim—. (7.14) 



It is not difficult to see that v is, in a proper space, a Lyapunov exponent (i.e. 
it measures the rate of divergence of two configurations) [214]. Consider two 
initial bi-infinite configurations cr(0) = (■•■, (T_2(0), o--i(O), (Ti(0), 0-2(0), ■■ •) 
ando-'(O) = (• • • , ct12(0), ct1i(0), a'i(O), ^2(0), ■ ■ ■) , with ^^(0) = ^^(O) for |i| < 

Nq, and their evolutions cr(t) and cr' {t). One can define a distance, \\5cr{ 
between cr{t) and cr (t), as follows: 



n=l 



where Sun — a'^ — On- With the above norm two systems can be arbitrarily 
close: one only needs A^o ^ 1- At this point it is possible to define the Lyapunov 
exponent as 

A=lim hm iln^^M. (7.I6) 

t-»oo ||5<T(0)||-»0 t ||5«t(0)|| 



Note that in (7.16) it has been implicitly taken the limit N — > 00. 




-100 -50 50 100 

i 

Fig. 35. Damage spreading analysis performed on a totalistic [Rule 10] r=2 cellular 
automaton with N = 200. At time t = a replica is initialized by flipping the value 
at the center of the lattice. 
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Noting that 5(T„(t) = for \n - No\ > Rt/2 ~ vt, while \5an{t)\ = 1 for 
\n — Nq\ < Rt/2 ~ vt, from the definition (7.15) one has: 

||(5<7(t)|| -2-^°+^*, (7.17) 

and therefore 

A = 't;ln2. (7.18) 



In other words, the hnear damage spreading in the physical space corresponds 
to an exponential growth in the norm (7.15). Oono and Yeung [167] stressed 
a conceptual (and practical) difficulty with the above approach. In systems 
with continuous states it is clear that by performing an infinitesimal change 
on a typical configuration one does not destroy the "typicality", i.e. the new 
initial condition will generate a trajectory belonging to the same attractor. 
On the contrary, it is not obvious that for a, however large, system with 
discrete states in a typical configuration a change of only one element gives 
another typical state. For instance, this seemingly innocent change can induce 
a jump among basins of attraction, so that the perturbed trajectory goes to a 
different attractor [223,16]. However, taking into account the above criticism, 
numerically one finds, for most the initial conditions, v > ior chaotic CA, 
and V — ior regular CA. 

We conclude this subsection mentioning a proposal, by Bagnoli et. al. [16], to 
introduce a Lyapunov exponent for cellular automata, defining it in analogy 
with continuous states dynamical systems. 

In this approach, the equivalent of an infinitesimal perturbation (as for the 
damage spreading analysis) is the difference between the system and one of its 
replicas in which one site has been flipped at time t — Then one formally 
introduces the Boolean derivatives, a sort of Jacobian of the rule, F- •, the 
elements of which are or 1. Here, for simplicity, we consider a generic nearest 
neighbor (r = 1) rule so that = for |i — j| > 2 and 

where the other nonzero terms are obtained by shifting the xor operation to i 
and i + 1 (respectively) . We recall that xor is the Boolean exclusive operation 
(i.e. OxorO = 0, IxorI = 0, OxorI = 1 and IxorO — 1). Of course as 
time goes on the initial perturbation spreads, i.e. new defects appear. As in 
continuous systems, one needs to maintain the perturbation "infinitesimal". 
One introduces a vector N (whose components, Ni, take integer values) which 
plays the role of the tangent vector. In order to mimic an infinitesimal pertur- 
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bation at the initial time one assumes A^i(O) = Sij, i.e. only one defect on the 
site j. The dynamics of Ni is ruled by the Boolean derivative i.e. 

N,{t + l)=J2FUt)N,{t). (7.19) 



Finally, putting |N(t)| = J2jNj{t), one can define the "Lyapunov exponent", 
Xb, of the cellular automaton as: 

Ab= hm ;^ln(|N(T)|). (7.20) 

1 — >oo _Z 



Now, in analogy with continuous systems, Ab < indicates an exponential 
decrease of the perturbation, while for As > the damage spreads. Just to 
give an example, if one considers the rule 150 of Wolfram classification, i.e. 
(F[0, 0, 1] = F[0, 1, 0] = F[l, 0, 0] = F[l, 1, 1] = 1 and otherwise) it is easy 
to see that F^j is a tridiagonal matrix with all the elements equal to 1 so that 
A = ln(3). For a generic rule one has to compute a suitable average over a 
long trajectory or on many initial configurations. 

The Lyapunov exponent, A^, has been demonstrated to be relevant in the syn- 
chronization problem [17] and allows for a qualitative characterization of the 
cellular automata in agreement with the classification proposed by Wolfram 
[16,17]. 



7.3.3 Entropies 

For cellular automata one can define a spatial/temporal entropy density by 
looking at the evolution of the elements in a subset Cl, of size L, of the 
system. Denoting with C{L,T) a "word" of spatial size L and time length T 
appearing in the time evolution of the elements in jCl, one defines the entropy 
of the subset Cl, 

hiL)^]im-^ Y: P{C{L,T))lnP{C{L,T)), (7.21) 



and then the spatio-temporal entropy density as 

/i^^ = lim ^h{L) . (7.22) 



105 



This entropy cannot be practically computed. A more accessible quantity is 
the temporal entropy: 

h^ = h{l)=]im -i Y: P{C{l,T))lnP{C{l,T)), (7.23) 

^C(l,T) 



i.e. the Shannon entropy of the time sequence of one element (cr„(0), cr„(l), ■ ■ ■). 
In principle, can depend on the site n and one can classify as nontrivial a 
system for which the majority of the elements have > [166]. An average 
measure of the "temporal disorder" is given by the spatial average {h'^). A 
systematic study of h{l) , h{2) , h{3) , . . . - although very difficult in practice 
- could give, in principle, relevant information on the spatial/temporal be- 
havior. A characterization of the spatial properties can be obtained studying, 
at a given time t, the spatial sequences. In practice, one studies C(L, 1) at 
increasing L: 

h^=]im -J Yl P{C{L,l))\nP{C{L,l)). (7.24) 



One can associate to a sort of "effective" dimension d = h^/\ii2 [223]. 
In a completely disordered cellular automaton configuration one has d — 1, 
as expected, while a homogeneous (or spatially periodic) configuration gives 
d = 0. 

From the definition of cellular automata (7.12) one easily sees that the value of 
ai{t) depends on sites at maximum distance r from i at the previous time step. 
This means that after T time steps, the value (Jj(t) can depend (at maximum) 
on sites at distance rT on both direction, so that the maximum speed for 
information propagation is r (i.e. the range of interaction). However, for many 
CA the actual velocity of information propagation, Vp, is less than r, i.e. (Ji{T) 
depends only on VpT < rT sites. By considering a simple construction (see 



2rT 
2vpT 




Fig. 36. Sketch of the dependence of temporal sequences on spatial ones. 
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Fig. 36) one can understand that the spatial and temporal entropies are related 
to each other by the inequality [223] : 

< 2vph^ , (7.25) 

where a good estimate of Vp can be given in terms of the damage spreading 
velocity (7.18) [223]. 

The possible scenario arising from (7.25) can be summarized as follows. One 
can have "spatial chaos" {h^ > 0) in absence of "temporal chaos" {h^ = 0), 
while the existence of "temporal chaos" requires not only a non zero spatial 
entropy but also the existence of a finite propagation velocity. This confirms 
somehow that the classifications of a CA as chaotic in terms of damage spread- 
ing velocity and entropy are related to each others. 

However, as stressed by Oono and Kohomoto [166] , the seemingly natural as- 
sumption of calling "turbulent" a cellular automaton for which one has > 
and (h'^) > is not correct in general. This is particularly clear by consid- 
ering a single direction shift imposed on a "frozen" disordered background. 
Nevertheless, in spite if this specific counterexample, the attempts based on 
entropic concepts, for the characterization of the irregular spatial and/or tem- 
poral behavior of systems with discrete states, in our opinion, are the most 
promising ones. In this context Casartelli and coworkers [47,48] introduced 
the concept of rational partitions in order to define a complexity measure for 
systems which can be reduced to Id CA. 

Let us conclude this Section with a brief discussion and comparison between 
the unpredictability which characterizes cellular automata evolution with re- 
spect to the one encountered in the context of continuous states dynamics, e.g. 
in coupled map lattices (see Sect. 4). The latter indeed seems to be the natu- 
ral candidate for such a comparison. We limit the discussion to 1-dimensional 
lattices with r — 1, i.e. CML and CA with nearest neighbor coupling. 

Let us now ask the amount of information we have to specify for knowing 
all the LT sites of spatial size L(< N) and temporal length T, as shown in 
Fig. 37. Since both CA and CML are ruled by a local deterministic dynamics 
one needs to specify the rule of evolution and the values of the L + 2(T — 1) 
states at the border, in black in Figure 37. Basically, one has to specify the 
initial conditions on the L sites and the "boundaries" (Ji{t) and (JL{t) for 
1 < t <T. But while for CA this specification unambiguously determines the 
LT values, for a chaotic CML this is not enough. Indeed, one has to specify the 
precision, e, with which he wants to know the LT values. Once specified e, one 
knows the necessary initial precision, cq, on the L + 2(T — 1) sites in black. A 
conservative estimate gives eo ~ eexp{—LT7iKs), where Hks is the entropy 
density defined in Eq. (4.3). This very simple argument suggests that the 
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L 

Fig. 37. The values of the sites in black together with the specification of the rule 
completely specify the values of the sites in white. 

main difTercnce between CA and continuous systems is the absence of "local" 
production of information, i.e. in CA the complexity only arises by the spatial 
propagation of information [97]. Nevertheless there exist counterexamples in 
which starting from simple initial configuration complex pattern are generated 
[222]. 

From this point of view it is interesting to consider the behavior of certain 
CMLs which, in spite of their continuous nature, seem to be rather similar to 
"chaotic" cellular automata. Indeed, it has been found that a class of stable 
(i.e. A < 0) CMLs [69,188] displays an unpredictable dynamics on times expo- 
nentially large with the system size. So that in the limit of infinite lattices they 
are completely unpredictable. Moreover, these CMLs have a finite velocity of 
propagation for initially localized disturbances (provided that the value of the 
disturbance was 0(1)) [188,190]. RecaUing the discussion of Sect. 4.6, we know 
that this cannot be predicted in terms of the comoving Lyapunov exponents, 
it is a fully non linear phenomenon. The strong analogies with "chaotic" CA 
have been furtherly explored in Ref. [188], where it has been proposed to clas- 
sify these CML as large memory cellular automata according to the behavior 
of their spatial and temporal entropy. 



8 The characterization of the Complexity and system modeling 

In the previous Sections we discussed the characterization of dynamical behav- 
iors when the evolution laws are known either exactly or with an uncertainty. 

On the other hand in experimental investigations only time records of some 
observable are typically available, and the equation of motions are not known. 
For the predictability problem, this latter case, at least from a conceptual 
point of view, can be treated in the same framework of when the evolution 
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laws arc known. Indeed, in principle, with the embedding technique one can 
reconstruct the phase space [209,1,2,114]. Nevertheless there are rather severe 
limitations in high dimensional systems [97] and even in low dimensional ones 
non trivial features appear in presence of noise [114]. 

In this Section we show that an entropic analysis at different resolution scales 
allows us for a pragmatic classification of a signal and gives suggestions for 
modeling of systems. In particular we illustrate, using some examples, how 
quantities such as the e-entropy or the FSLE can display a subtle transi- 
tion from the large to the small scales. A negative consequence of this is the 
difficulty in distinguishing, only from data analysis, a genuine deterministic 
chaotic system from one with intrinsic randomness [51]. On the other hand, the 
way the e-entropy or Finite Size Lyapunov Exponent depends on the (resolu- 
tion) scale, allows us for a pragmatic classification of the stochastic or chaotic 
character of the signal, and this gives some freedom in modeling the system. 

8.1 How random is a random number generator? 

It is rather natural to wonder about the "true character" of the number 
sequence {xi,X2,---) obtained with a (pseudo) random number generator 
(PRNG) on a computer. One would like to have a sequence with a random 
character; on the other hand, one is forced to use deterministic algorithms to 
generate {xi,X2, ■ ■ ■)■ This subsection is mainly based on the paper [116]. A 
simple and popular PRNG is the multiplicative congruent one [193]: 



with an integer multiplier A^i and modulus The {zn} are integer numbers 
and one hopes to generate a sequence of random variables {x„}, which are 
uncorrelated and uniformly distributed in the unit interval. A first problem 
one has to face is the periodic nature of (8.1), because of its discrete character 
(see Sect. 7). In practice one wants to fix A''i and A^2 in such a way to maximize 
this period. Note that the rule (8.1) can be interpreted as a deterministic 
dynamical system, i.e. 



which has a uniform invariant measure and a KS entropy hxs = A = InA^i. 
When imposing the integer arithmetics of Eq. (8.1) onto this system, we are, 
in the language of dynamical systems, considering an unstable periodic orbit 
of Eq. (8.2), with the particular constraint that, in order to achieve the period 



Zn+i = NiZn mod N2 



(8.1) 



Xn+i = NiXn mod 1 , 



(8.2) 
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N2 — I (i.e. all integers < N2 should belong to the orbit of Eq. (8.1)) it has to 
contain all values k/N2, with k = 1,2, ■ ■ ■ , N2 — 1. Since the natural invariant 
measure of Eq. (8.2) is uniform, such an orbit represents the measure of a 
chaotic solution in an optimal way. Every sequence of a PRNG is characterized 
by two quantities: its period T and its positive Lyapunov exponent A, which is 
identical to the entropy of a chaotic orbit of the equivalent dynamical system. 
Of course a good random number generator has a very large period, and as 
large as possible entropy. 

It is natural to ask how this apparent randomness can be reconciled with 
the facts that (a) the PRNG is a deterministic dynamical systems (b) it is a 
discrete state system. 

If the period is long enough on shorter times one has to face only point (a). In 
the following we discuss this point in terms of the behavior of the e-entropy, 
h{e) (sec Sect. 3.5). It seems rather reasonable to think that at a high reso- 
lution, i.e. e < 1/A^i, one should realize the true deterministic chaotic nature 
of the system and, therefore, h{e) ~ hxs = InA^i. On the other hand for 
e > 1/A^i one expects to observe the "apparent random" behavior of the 
system, i.e. h{e) ~ ln(l/e). When the spatial resolution is high enough so 




0.001 0.01 0.1 



e 

Fig. 38. The e-entropies, hm{e), at varying the embedding dimension m for the 
multipUcative congruential random number generator Eq. 8.1 for different choices 
of Ni and N2. This figure has been taken from Ref. [116]. 

that every point of this periodic orbit is characterized by its own symbol, 
then, for arbitrary block length m, one has a finite number of m-words whose 
probabilities are different from 0. Therefore, the block entropy (3.44) is 
m-independent and hm — 0. 
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In Fig. 38 it is shown the behavior of h„i{e), computed on sequences of length 
60000 of the PRNG with three different pairs (A^i, A^z) chosen to be (7^,2^^), 
(2,494539), and (107,31771). The first one is optimal and no deviation from 
the stochastic behavior is visible. The second one has a small pseudo-entropy, 
and this is seen by the saturation of all hm{€) at InA^i = In 2, and the last 
one has large entropy but a rather short period, so that all hmi^) drop to zero 
for some e^, where becomes dramatically larger for increasing m (strong 
fluctuations arise from the fact that data are confined to a grid of spacing 
1/31771). 



8.2 High dimensional systems 



Now we discuss high-dimensional systems that show non-trivial behavior at 
varying the resolution scales. Olbrich et al. [165] analyzed an open flow system 
described by unidirectionally coupled map lattice: 

xj{t + 1) = (1 - a)f{xj+^{t)) + axj{t) (8.3) 



where j = 1, . . . , iV denotes the site of a lattice of size A^, t the discrete time 
and a the coupling strength. A detailed numerical study (also supported by 
analytical arguments) of the e-entropy hmi^) at different e, in the limit of small 
coupling, gives the following scale-dependent scenario: for 1 > e > a there is 
a plateau h{e) ~ where is the Lyapunov exponent of the single map 
x{t + l) = f{x{t)). For a > e > another plateau appears at h{e) ~ 2A5, and 
so on: for a'"~^ > e > (t" one has h{e) ~ nXg (see Fig. 39). Similar results hold 
for the correlation dimension which increases step by step as the resolution 
increases, showing that the high- dimensionality of the system becomes evident 
only as e — 0. Therefore one understands that the dynamics at different scales 
is basically ruled by a hierarchy of low-dimensional systems whose "effective" 
dimension ne//(e) increases as e decreases [165]: 



n. 



eff 



" MIA) " 

_ln(l/<7). 



where [. . .] indicates the integer part. In addition, for a given resolution e, it 
is possible to find a suitable low-dimensional noisy system (depending on e) 
which is able to mimic Xi(t) given by Eq. (8.3). It is interesting to note that, on 
an extended range of values of e (e > cr^), h{e) can be roughly approximated 
as log-periodic fluctuations around 

h{e) ~ In - (8.5) 
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Fig. 39. /im(e) for the system (8.3 where f{x) = 2|l/2 — \x — 1/2|| is the tent map 
and cr = 0.01. The horizontal hnes indicate the entropy steps which appears at 
decreasing e. The e-entropy is computed with the Grassberger-Procaccia method 
[92]. For further details see Ref. [165] 

i.e. the typical behavior of a stochastic process. Of course for e < one has 
to realize that the system is deterministic and h{e) = 0{NXs). 

Let us now briefly reconsider the issue of the macroscopic chaos, discussed in 
Sect. 4.7. The main result can be summarized as follows: 

• at small e 1/V^), where is the number of elements, one recovers the 
"microscopic" Lyapunov exponent^, i.e. A(e) ^ Xmicro 

• at large e {':^ I/a/ZV) one observes another plateau A(e) ^ Xmacro which can 
be much smaller than the microscopic one. 

The emerging scenario is that at a coarse-grained level, i.e. e ^ 1/\/N, the 
system can be described by an "effective" hydro-dynamical equation (which in 
some cases can be low-dimensional), while the "true" high-dimensional char- 
acter appears only at very high resolution, i.e. 



^ From hereafter we use the same symbol e both for the FSLE and the e-entropy 
in order to make a direct comparison between the two quantities 
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8.3 Diffusion in deterministic systems and Brownian motion 



Consider the following map which generates a diffusive behavior on the large 
scales [200]: 



xt+i = [xt] +F{xt- [xt]), 



1.6) 



where [xt] indicates the integer part of Xt and F{y) is given by: 



{2 + a)y if y e [0,l/2[ 

{2 + a)y-{l + a) if y e [1/2,1]. 



The largest Lyapunov exponent A can be obtained immediately: A ~ ln|F'|, 
with F' — dF/dy —2 + a. One expects the following scenario for h{e): 



h{e) A for e < 1, 
D 



h{e) oc 



for e > 1, 



(8.8) 
(8.9) 



where D is the diffusion coefficient, i.e. 



{{xt — xof) K.2 D t for large t. 



(8.10) 




Fig. 40. The map F{x) (8.7) for a = 0.4 is shown with superimposed the approxi- 
mating (regular) map G{x) (8.11) obtained by using 40 intervals of slope 0. 

Consider now a stochastic system, namely a noisy map 

xt+i = [xt] +G{xt- [xt]) + ar]t, (8.11) 
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where G{y), as shown in Fig. 40, is a piece wise hnear map which approximates 
the map F{y), and rjt is a stochastic process uniformly distributed in the 
interval [—1, 1], and no correlation in time. When \dG/dy\ < 1, as is the case 
we consider, the map (8.11), in the absence of noise, gives a non-chaotic time 
evolution. 
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Fig. 41. A(e) versus e obtained with the map F{y) (8.7) with a = 0.4 (o) and with 

the noisy (regular) map (□) (8.11) with 10000 intervals of slope 0.9 and a = 10~^. 
The straight lines indicates the Lyapunov exponent A = In 2.4 and the diffusive 
behavior A(e) ~ e~^. 



Now we compare the finite size Lyapunov exponent for the chaotic map (8.6) 
and for the noisy one (8.11). In the latter the FSLE has been computed using 
two different realizations of the noise. In Fig. 41 we show A(e) versus e for 
the two cases. The two curves are practically indistinguishable in the region 
e > (7. The differences appear only at very small scales e < a where one has 
a A(e) which grows with e for the noisy case, remaining at the same value for 
the chaotic deterministic case. 

Both the FSLE and the (e, r)-entropy analysis show that we can distinguish 
three different regimes observing the dynamics of (8.11) on different length 
scales. On the large length scales e > 1 we observe diffusive behavior in both 
models. On length scales a < e < 1 both models show chaotic deterministic 
behavior, because the entropy and the FSLE are independent of e and larger 
than zero. Finally on the smallest length scales e < cr we see stochastic behav- 
ior for the system (8.11) while the system (8.6) still shows chaotic behavior. 
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8-4 On the distinction between chaos and noise 



The above examples show that the distinction between chaos and noise can 
be a high non trivial task, which makes sense only in very peculiar cases, e.g., 
very low dimensional systems. Nevertheless, even in this case, the entropic 
analysis can be unable to recognize the "true" character of the system due to 
the lack of resolution. Again, the comparison between the diffusive map (8.6) 
and the noisy map (8.11) is an example of these difficulties. For a < e < 1 
both the system (8.6) and (8.11), in spite of their "true" character, will be 
classified as chaotic, while for e > 1 both can be considered as stochastic. 

In high-dimensional chaotic systems, with N degrees of freedom, one has typ- 
ically /i(e) = Hks ~ 0{N) for e < €c (where ec — > as A?" — > oo) while for 
e > Co h{e) decreases, often with a power law [89] . Since also in some stochastic 
processes the e-entropy obeys a power law, this can be a source of confusion. 

These kind of problems are not abstract ones, as a recent debate on "micro- 
scopic chaos" demonstrates [90,72,98]. The detection of microscopic chaos by 
data analysis has been recently addressed in a work of Gaspard et al. [90]. 
These authors, from an entropic analysis of an ingenious experiment on the 
position of a Brownian particle in a liquid, claim to give an empirical evidence 
for microscopic chaos. In other words, they state that the diffusive behavior 
observed for a Brownian particle is the consequence of chaos at a molecular 
level. Their work can be briefly summarized as follows: from a long (fsi 1.5 x 10^ 
data) record of the position of a Brownian particle they compute the e-entropy 
with the Cohen-Procaccia method [61] (Sect. 3) from which they obtain: 



where D is the diffusion coefficient. Then, assuming that the system is deter- 
ministic, and making use of the inequality h{e > 0) < h^St they conclude that 
the system is chaotic. However, their result does not give a direct evidence that 
the system is deterministic and chaotic. Indeed, the power law (8.12) can be 
produced with different mechanisms: 

(1) a genuine chaotic system with diffusive behavior, as the map (8.7); 

(2) a non chaotic system with some noise, as the map (8.11), or a genuine 
Brownian system; 

(3) a deterministic linear non chaotic system with many degrees of freedom 
(see for instance [158]); 

(4) a "complicated" non chaotic system as the Ehrenfest wind-tree model 
where a particle diffuses in a plane due to collisions with randomly placed, 
fixed oriented square scatters, as discussed by Cohen et al. [72] in their 



h{e) 



D 
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comment to Ref. [90]. 

It seems to us that the weak points of the analysis in Ref. [90] are: 

a) the explicit assumption that the system is deterministic; 

b) the limited number of data points and therefore limitations in both reso- 
lution and block length. 

The point (a) is crucial, without this assumption (even with an enormous 
data set) it is not possible to distinguish between 1) and 2). One has to say 
that in the cases 3) and 4) at least in principle it is possible to understand 
that the systems arc "trivial" (i.e. not chaotic) but for this one has to use a 
huge number of data. For example Cohen et al. [72] estimated that in order 
to distinguish between 1) and 4) using realistic parameters of a typical liquid, 
the number of data points required has to be at least ~ 10^^. 

Concluding, we have the apparently paradoxical result that "complexity" helps 
in the construction of models. Basically, in the case in which one has a variety 
of behaviors at varying the scale resolution, there is a certain freedom on the 
choice of the model to adopt. In Sect. 8.3 one can see that, for some systems, 
the behavior at large scales can be realized both with chaotic deterministic 
models or suitable stochastic processes. From a pragmatic point of view, the 
fact that in certain stochastic processes h{e) ~ e^" can be indeed extremely 
useful for modeling such high- dimensional systems. Perhaps, the most relevant 
case in which one can use this freedom in modeling is the fully developed 
turbulence whose non infinitesimal (the so-callcd incrtial range) properties 
can be successfully mimicked in terms of multiaffine stochastic process (see 
Ref. [29,4] Sect. 5.5 and Appendix D). 



9 Concluding Renicirks 

The guideline of this review has been how to interpret the different aspects of 
the predictability of a system as a way to characterize its complexity. 

We have discussed the relation between the Kolmogorov-Sinai entropy and the 
algorithmic complexity (Sect. 2). As clearly exposed in the seminal works of 
Alekseev and Yakobson [5] and Ford [79,80], the time sequences generated by 
a system with sensitive dependence on initial conditions have non-zero algo- 
rithmic complexity. A relation exists between the maximal compression of a 
sequence and its KS-entropy. Therefore, one can give a definition of complex- 
ity, without referring to a specific description, as an intrinsic property of the 
system. 
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In presence of intrinsic randomness (Sect . 6.3), one can introduce two different 
Lyapunov exponents, A^- in the case of trajectories with the same reahzation of 
noise and K^r for different reahzations. In general A^. and K^^ do not coincide 
and characterize different aspects of the system. Both quantities have their 
own relevance, the comparison between Ao- and Kcr has shown to be useful in 
the understanding of apparently intricate phenomena, such as noise-induced 
order and noise-induced instability. 

As an example of system with many degrees of freedom and characteristic 
times scales, we investigated fully developed turbulence (Sect. 5). In this case 
the Lyapunov exponent and the KS-entropy are somehow of limited relevance 
because they only characterize small scales properties. On the other hand there 
exist suitable generalizations - the finite size Lyapunov exponent, A(e), and 
e-entropy, h{e) - which characterize the predictability properties at different 
scales. The scaling of the predictabihty time with the resolution e, A(e) ~ e~^, 
has an algorithmic correspondence in the behavior of the e-entropy of the 
signal measured in one point, h{t) ~ e~^. In the words of Lorenz, one can say 
that the butterfly effect is not so terrible for e-resolution in the inertial range. 

Complexity in a system can also manifest in the spatial properties as, for 
example, in open flows with convcctive chaos but with negative Lyapunov 
exponents (Sect. 4). The presence of convective chaos implies a sensitivity on 
the boundary conditions. An uncertainty, Sxq, on the boundary condition is 
exponentially amplified with the distance, n, from the boundary as 5xn ~ 
SxqC^'^. The "spatial" Lyapunov exponent F is related with the comoving 
Lyapunov exponent and gives a characterization of the spatial "complexity" . 

The study of these different aspects of predictability constitutes a useful 
method for a quantitative characterization of "complexity", suggesting the 
following equivalences: 



The above point of view, based on dynamical systems and information theory, 
quantifies the complexity of a sequence considering each symbol relevant but it 
does not capture the structural level. Let us clarify this point with the following 
example. A binary sequence obtained with a coin tossing is, from the point 
of view adopted in this review, complex since it cannot be compressed (i.e. it 
is unpredictable). On the other hand such a sequence is somehow trivial, i.e. 
with low "organizational" complexity. It would be important to introduce a 
quantitative measure of this intuitive idea. The progresses of the research on 




Complex 



Uncompressible 



Unpredictable 



117 



this intriguing and difficult issue are still rather slow. We just mention some of 
the most promising proposals as the logical depth [21] and the sophistication 
[130], see Ref. [13]. 

As a final question one can wonder what can one learn by the presented mate- 
rial for practical prediction problems (e.g. weather forecast). The main lesson 
concerns the framework and limitations about the possible well posed ques- 
tions in prediction and modeling. The first relevant fact, now well established, 
is that in presence of deterministic chaos one cannot hope to reach an arbi- 
trary accuracy in prediction by merely refining the model. A less recognized 
aspect is that the Lyapunov exponent is usually not sufficient to characterize 
the limits of predictability in real situations. An appropriate generalization of 
the Lyapunov exponent is necessary to account for the large scale properties. 
Moreover in weather forecast the predictability time, which is typically of 5 
days, may be as little as 2 days or as much as 10 days [211]. Thus, simply 
quoting an average value does not give a satisfactory answer. At a more con- 
ceptual level, one has severe limitations in distinguish between deterministic 
or stochastic nature of systems displaying complex behavior. This implies a 
certain freedom in the choice of the details of the model, in particular whether 
to adopt a deterministic or a stochastic model. 
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A On the computation of the Finite Size Lyapunov Exponent 

This Appendix is devoted to the methods for computing the Finite Size Lya- 
punov Exponent. As we will see there are mainly three possible ways to com- 
pute the FSLE. Let us start with a modification of the standard technique for 
computing the largest Lyapunov exponent[20,221]. 

Suppose to integrate on a computer the equations of motion of a system. After 
a long integration time, in order for the motion to settle onto the attractor 
of the system, wc introduce a very small perturbation, i.e. we consider the 
"reference" trajectory x(0), which is supposed to be on the attractor, and 
generate a "perturbed" trajectory starting from x'(0) = x(0) -|-5x(0). We need 
the perturbation to be initially very small in some chosen norm 5{t — 0) — 
||(5x(t = 0)11 = Smin -C 1. Then, in order to study the perturbation growth 
through different scales, one defines a set of thresholds 5„, e.g.: 5„ = ^q'"^"'^ 
with 1 ^ Sq ^ Smin and n — 0, . . . , N. To avoid saturation on the maximum 
allowed separation (i.e. the attractor size) one has to choose 5n < (| |x — y 1 1),^ 
with X, y generic points on the attractor. Note that r should be larger than 
1 but not too large in order to avoid interferences of different length scales. 
Typically, one chooses r = 2 or r = ^/2. 

In order to measure the perturbation growth rate at scale 5„, one lets the 
system to evolve from Smin up to the desired scale 5n ensuring the perturbation 
to be on the attractor and aligned along the maximally expanding direction. 
After 6n is reached, one computes the first time, ri(5„, r), to reach the following 
threshold, 5^+1; and after that the perturbation is rescaled to keeping 
the direction x' — x constant. This procedure is repeated Af times for each 
thresholds obtaining the set of the doubling times {Ti(5„, r)} ioi i — 1, . . . , jV 
error-doubling experiments. Now if we introduce the effective doubling rates: 

7i(^n,?^) = ^P — rlnr, (A.l) 

we can define their time averages as the the effective LEs on the scale 
Therefore, we have 

MK) - MK, r)),^^ / 7* ^ If ^ . (A.2) 

where (t((5„, r))e = E n/Af and T = Ei n. 

To obtain Eq. (A.2) we assumed the distance between the two trajectories to 
be continuous in time. This is not true for maps or for discrete sampling in 
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time and the method has to be shghtly modified. In this case the doubhng 
time, r((5„,r), is defined as the minimum time such that S{r) > rSn- Because 
now 5{t) is a fiuctuating quantity, from (A. 2) we have 



Let us stress some points. 

The computation of the FSLE is not more expensive than the one of the 
Lyapunov exponent by standard algorithm. One has simply to integrate two 
copies of the system (or two different systems for second kind predictability) 
and this can be done without particular problems. 

At difference with A, X{6) may also depend on the norm one chooses. This 
fact, apparently disturbing, is however physically reasonable: when one looks 
at the non linear regime, for instance, for the predictability problem the answer 
may depend on the involved observable. A similar problem appears in infinite 
dimensional system where the norms are not equivalent [127]. 

A possible problem with the above described method is that we have implicitly 
assumed that the statistically stationary state of the system is homogeneous 
with respect to finite perturbations. Actually one may plausibly expect the 
attractor to be fractal, i.e., not at all equally dense at all distances, this may 
cause an incorrect sampling of the doubling times at large 

A possible way to overcome such a problem is to compute the FSLE avoiding 
to rescalc the perturbation at finite This can be accomplished by the 
following modification of the previous method. One defines the thresholds 
{5n} and initializes the perturbation at 5min <^ Sq as before. Then one lets 
the system to reach the first threshold, 5o. Hence, one starts to measure the 
doubling time r(5„,r) following the perturbation growth from 60 up to Sn- 
In practice, one register the time r(5„,r) for going from 5„ to 6n+i for each 
value of n. The evolution of the error from the initial value 6min to the largest 
threshold carries out a single error-doubling experiment. When the largest 
threshold, 5^ has been reached the "perturbed" trajectory is rescaled at the 
initial distance, ^,„,;n, with respect to the "reference" trajectory and one starts 
another experiment measuring a second set of doubling times, {T2{Sn,r)}. 
The procedure is then repeated J\f times to have statistics. In this way one 
obtains the set of the doubling times {ri(5„, r)} for i = 1, . . . , jV. The FSLE 
is finally obtained by using Eq. (A. 2) or Eq. (A. 3), which are accurate also 
in this case, according to the continuous time and discrete time nature of 
the system respectively. One understands that with this method, since finite 
perturbations are realized by the dynamics (i.e. the perturbed trajectory is on 
the attractor) and not imposed by hand, the problems related to the attractor 
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inhomogeneity are not present. 



In any case, in most numerical experiments, one does not find significant 
differences between the two numerical methods. 

A further possibility to compute the FSLE is to remove the threshold condition 
used for defining T(5„,r) and simply compute the average error growth rate 
at every time step. In other words, at every time step At in the integration, 
the perturbed trajectory x'{t) is rescaled to the original distance S, keeping 
the direction x — x' constant. The FSLE is then obtained by the average of 
the one-step exponential divergence: 



which, if non negative, is equivalent to the definition (A. 2). Let us note that 
the above procedure is nothing but the finite scale version of the usual al- 
gorithm of Benettin et al. [20] for the LE. The one-step method (A.4) can 
be, in principle, generalized to compute the sub-leading finite-size Lyapunov 
exponent following the standard ortho-normalization method [20]. One intro- 
duces k perturbed trajectories x*^^), . . . , x'^''^ each at distance 6 from x and 
such that x^*') — x are orthogonal each to the others. At every time step, any 
difference x^*^) — x is rescaled at the original value and orthogonalized, while 
the corresponding finite size Lyapunov exponent is accumulated according to 



Here we have again the problem of the implicitly assumed homogeneity of 
the attractor, but also a problem of isotropy when we re-orthogonalize the 
perturbations. We note that this could be a more serious problem. 



B The multifractal model of turbulence 

The multifractal model of turbulence [177,172,84] assumes that the velocity 
has a local scale-invariance, i.e. it does not have a unique scaling exponent h 
such that 6v£ ~ but a continuous spectrum of exponents, each of which 
belonging to a given fractal set. In other words, in the inertial range one has 



if a; e Sh, and Sh is a fractal set with dimension D{h) and h G {hmim hmax)- 
The probability to observe a given scaling exponent h at the scale i is thus 




(A.4) 



(A.4). 



Svi{x) ~ r , 



(B.l) 
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Pt{h) ~ so the scaling of the structure function assumes the form: 

hmax 

Sp{t)^{5vf)r. J &e-''^^^d.h^6- , (B.2) 

where in the last equality, being ^ <S 1, a steepest descent estimation gives 

Cp = min {hp + 2,- D{h)} = h*p + 3- D{h*) (B.3) 

h 



where h* = h*{p) is the solution of the equation D'{h*{p)) — p. The Kol- 
mogorov "4/5" law [84] 

^3(^) = -Ui (B.4) 
o 



imposes — ^ which implies that 

3h + 2-D{h)<Q, (B.5) 

the equality is realized by h*{3). The Kolmogorov similarity theory corre- 
sponds to the case of only one singularity exponent h = 1/3 with D{h — 
1/3) = 3. 

A nontrivial consequence of the intermittency in the turbulent cascade is the 
fluctuations of the dissipative scale which implies the existence of an interme- 
diate region between the inertial and dissipative range [83]. The local dissipa- 
tive scale Id is determined by imposing the effective Reynolds number to be 
of order unity: 

Re{in) = ^ ~ 1 , (B.6) 

therefore the dependence ol Id on his thus 

tD{h) ~ LRe-^ (B.7) 



where Re = Re{L) is the large scale Reynolds number. The fluctuations of 
Id modifles the evaluation of the structure functions (B.2): for a given the 
saddle point evaluation of (B.2) remains unchanged if, for the selected expo- 
nent h*{p), one has ^D{h*{p)) < i. If, on the contrary, the selected exponent is 
such that (■D{h*{p)) > i the saddle point evaluation is not consistent, because 
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at scale I the power-law scaling (B.l) is no longer valid. In this intermedi- 
ate dissipation range [83] the integral in (B.2) is dominated by the smallest 
acceptable scaling exponent given by inverting (B.7), and the structure 
function of order p a pseudo-algebraic behavior, i.e. a power law with expo- 
nent ph{t) -I- 3 — D{h{£)) which depends on the scale i. Taking into account 
the fluctuations of the dissipative range, one has for the structure functions 

M'^) ~ I ^He)p+s-D(He)) jf ^^^^^.^ < ^ < iD{h*ip)) . ^ 



A simple calculation [83,84] shows that it is possible to find a universal descrip- 
tion valid both in the inertial and in the intermediate dissipative ranges. Let 
us discuss this point for the energy spectrum E{k). Introducing the rescaled 
variables 

= and e = ^ (B.9) 

^ ^ lni?e lni?e ^ ^ 



one obtains the following behavior 



-(1 + C2)^ for^< 



The prediction of the multifractal model is that ln£'(fc)/lni?e is an universal 
function of lnA;/lni?e. This is in contrast with the usual scaling hypothesis 
according which In E{k) should be a universal function of \n{k/kD))- The 
multifractal universality has been tested by collapsing energy spectra obtained 
from turbulent flow in a wide range of Re [88], see also [30]. 



C How to compute the e-entropy with exit times 



The approach based on exit times differs from the usual one (see Sect. 3.5) in 
the procedure to construct the coding sequence of the signal at a given level of 
accuracy [3]. Indeed an efficient coding procedure reduces the redundancy and 
improves the quality of the results. The method here discussed is particularly 
suited for computing the e-entropy in processes in which many scales are 
excited as, e.g., in turbulence [3,4]. 

The coding of a signal, x{t), by the exit-time approach is the following. Given 
a reference starting time t = to, one measures the first exit-time, ti, from a 
cell of size e, i.e. the first time necessary to have \x{to + ti) — x{to)\ > e/2. 
Then one restarts from the time t — to + ti to look for the next exit-time t2, 
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i.e., the first time such that \x{to + ti + 3^2) — x{to + > e/2 and so on. 
Finally one obtains a sequence of exit-times, {tj(e)}, and one also records the 
labels ki = ±1, which distinguish the direction of the exit (up or down out of 
a cell). 

At the end of this construction, the trajectory is coded without ambiguity, 
with the required accuracy e, by the sequence {{ti, ki), i = 1, . . . , M}, where 
M is the total number of exit-time events observed during the total time 
T. Now, performing a coarse-graining of the possible values assumed by t{e) 
with a resolution time r^, we accomplish the goal of obtaining a symbolic 
sequence. One now studies the "exit-time words", flf^ , of various lengths n: 
nf (e, Tr) = {{r]i, ki), {r]i+i, ki+i), {r]i+N^i, h+N^i)), where r]j labels the cell 
(of width Tr) containing the exit-time tj. From the probabilities of these words 
one evaluates the block entropies (2.18) at the given time resolution, H^{e, Tr), 
and then the exit-time (e, r^) -entropies: 

h''{e,Tr) = lim H^+,{e,Tr) - H^{e,Tr) . (C.l) 

N—^0O 

The limit of infinite time-resolution gives us the e-entropy per exit, i.e.: 

h^(e) ^ lim h^(e,Tr) . (C.2) 

Tr— >0 



The link between h^{e) and the e-entropy (3.45) can be obtained as follows. 
We note that there is a one-to-one correspondence between the (exit-time)- 
histories and the (e, r)-histories (in the limit r ^ 0) originating from a 
given e-cell. The Shannon-McMillan theorem [121] assures that the number 
of the typical (e, r)-histories of length N, Af{e, N), is such that: liij\f{e, N) ~ 
h{e)NT — h{e)T. For the number of typical (exit-time)-histories of length M, 
M{e, M), we have: \nM{e, M) ~ hP'{e)M. If we consider T = M{t(e)), where 
(t(e)) = 1/M Y.iti'ti, we must obtain the same number of (very long) his- 
tories. Therefore, from the relation M — T/{t{e)) we obtain finally for the 
e-entropy per unit time: 

Mh«(,) _ h»(,.T,:) 



Where the last equality is valid at least for small enough Tr [3]. In most of 
the cases, the leading e-contribution to h{e) in (C.3) is given by the mean 
exit-time {t{e)) and not by h^{e,Tr). Anyhow, the computation of hP[e,Tr) is 
compulsory in order to recover, e.g., a zero entropy for regular (e.g. periodic) 
signals. 

One can easily estimate an upper and a lower bound for h{e) which can be 
computed in the exit time scheme [3]. We use the following notation: for given 
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e and Tr, h^{e,Tr) = h^{{r]i,ki}), and we indicate with h^{{ki}) and h^i{rii}) 
respectively the Shannon entropy of the sequence {fcj} and By applying 
standard results of information theory [201] one obtains the inequalities (see 
[3,4] for more details): 

h'^iiki}) < h'^iirji, k}) < h'^iirji}) + h'^iiki}). (C.4) 



Moreover, h^{{r]i}) < H^{{rii}), where H^{{r]i}) is the one-symbol entropy of 
{fji}, (i.e. the entropy of the probability distribution of the exit-times measured 
on the scale r^) which can be written as 

^r(M) = c(6)+ln(i^) , 

where c(e) = — / p{z) lnp{z)dz, and p{z) is the probability distribution func- 
tion of the rescaled exit-time z{e) — t{e)/{t{e)). Finally, using the previous 
relations, one obtains the following bounds for the e-entropy: 

iiwr - m ■ 



Note that such bounds are relatively easy to compute and give a good estimate 
of h{e). In particular, as far as the scaling behavior of /i(e) is concerned, the 
exit-time method allows for very efficient and good estimates of the scaling 
exponent. The reason is that at fixed e, (t(e)) automatically selects the typ- 
ical time at that scale. Consequently, it is not necessary to reach very large 
block sizes - at least if e is not too small. So that the leading contribution is 
given by (t(e)), and h^{e,Tr) introduces, at worst, a sub-leading logarithmic 
contribution h^{e,Tr) ~ ln((i(e))/Tr) (see Eq. (C.5)). 

In Ref. [3,4] one can found the details of the derivation and some applications. 



D Synthetic signals for turbulence 



In this Appendix we recall some recently introduced methods for the genera- 
tion of multi-affine stochastic signals [27,29], whose scaling properties are fully 
under control. The first step consists in generating a 1-dimensional signal, and 
the second in decorating it such as to build the most general (d-|-l) -dimensional 
process, v{'x,t), with given scaling properties in time and in space. 
For the 1-dimcnsional case there are at least two different kind of algorithms. 
One is based on a dyadic decomposition of the signal in a wavelet basis with a 
suitable assigned series of stochastic coefficients [27]. The second is based on 
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a multiplication of sequential Langevin-processes with a hierarchy of different 

characteristic times [29]. 

The first procedure suits particularly appealing for modeling of spatial turbu- 
lent fluctuations, because of the natural identification between wavelets and 
eddies in the physical space. The second one looks more appropriate for mim- 
icking the turbulent time evolution in a fixed point of the space. 

D. 1 Reproducing the spatial properties or the temporal ones 

A non-sequential algorithm for 1-dimensional multi-affine signal in [0, 1], v{x), 
can be introduced as [27]: 

n=l k=l V *n / 

where we have a set of reference scales f„ = 2"" and ^{x) is a wavelet-like 
function [77], i.e. of zero mean and rapidly decaying in both real space and 
Fourier-space. The signal v{x) is built in terms of a superposition of fluctu- 
ations, ip{{x — Xn,k)/(-n) of characteristic width ^„ and centered in different 
points of [0, 1], Xn,k — (2/c + l)/2"+^. In [29] it has been proved that provided 
the coefficients a„,fc are chosen by a random multiplicative process, i.e. the 
daughter is given in terms of the mother by a random process, ttn+i^k' = ^cin,k 
with X a random number identical, independent distributed for any {n,k}, 
then the result of the superposition is a multi-affine function with given scaling 
exponents, namely: 

{{\v{x + K)-v{x)\P)) , 

with ({p) = —p/2 — log2(X^) and < i? < 1. In this Appendix (■) indicates 
the average over the probability distribution of the multiplicative process. 

Besides the rigorous proof, the rationale for the previous result is simply that 

due to the hierarchical organization of the fluctuations, one may easily see 
that the term dominating the expression of a velocity fluctuation at scale R, 
in (D.l) is given by the couple of indices {n,k} such that n ~ log2{R) and 
X ~ Xn,k, i-e. v{x + R) — v{x) ~ an,k- The generalization (D.l) to d-dimension 
is given by: 

^W = E E (^n,k^[ . j , 

n=l k=l \ / 

where now the coefficients {an,k} ^-re given in terms of a d-dimensional dyadic 
multiplicative process. 

On the other hand, as previously said, sequential algorithms look more suit- 
able for mimicking temporal fluctuations. Let us now discuss how to construct 
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these stochastic multi-affine fields. With the apphcation to time-fluctuations 
in mind, we wiU denote now the stochastic 1-dimensional functions with u{t). 
The signal u{t) is obtained by a superposition of functions with different char- 
acteristic times, representing eddies of various sizes [29] : 

N 

uit) = Y.Un{t) . (D.2) 
n=l 



The functions Un{t) are defined by the multiplicative process 

Un{t) = gn{t)xi{t)x2{t) . . . Xn{t) , (D.3) 



where the g„{t) are independent stationary random processes, whose correla- 
tion times are supposed to be t„ = (^n)", where a — 1 — h (i.e. t„ are the 

eddy-turn-over time at scale in) in the quasi-Lagrangian frame of reference 
[152] and a = 1 if one considers u(t) as the time signal in a given point, and 
(fl'n) ~ {^nY^i where h is the Holder exponent. For a signal mimicking a turbu- 
lent flow, ignoring intermittency, we would have h — 1/3. Scaling will appear 
for all time delays larger than the UV cutoff tn and smaller than the IR cutoff 
Ti. The Xj{t) are independent, positive deflned, identical distributed random 
processes whose time correlation decays with the characteristic time r,-. The 
probability distribution of Xj determines the intermittency of the process. 

The origin of (D.3) is fairly clear in the context of fully developed turbulence. 
Indeed we can identify Un with the velocity difference at scale in and Xj with 
(£j/£j_i)^/^, where Sj is the energy dissipation at scale Ij [29]. 

The following arguments show, that the process deflned according to (D.2, D.3) 
is multi-affine. Because of the fast decrease of the correlation times Tj — (ij)"', 
the characteristic time of Un{t) is of the order of the shortest one, i.e., r„ = 
(£„)". Therefore, the leading contribution to the structure function Sq{T) ~ 
{{\u{t + T) —u{t)\'^)) with T ~ T„ stems from the n-th term in (D.2). This can 
be understood nothing that in u{t + r) — u{t) — J2k=i[''^k{t + ''■)— Uk{t)] the 
terms with k < n are negligible because Uk{t -|- r) ~ Uk{t) and the terms with 
k > n are sub-leading. Thus one has: 

hq log2(j:'^) 

S,{Tn) - {\Un\') - {\gn\'){xT ^n" " (D.4) 



and therefore for the scaling exponents: 



C, = ^-i^^. (D.5) 

a a 
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The limit of an affine function can be obtained when all the equal to 1. 

A proper proof of these result can be found in [29]. 

Let us notice at this stage that the previous "temporal" signal for a = 1—h is a 
good candidate for a velocity measurements in a Lagrangian, co-moving frame 
of reference [152]. Indeed, in such a reference frame the temporal decorrelation 
properties at scale £„ are given by the eddy-turn-over times r„ = {£nY~^- On 
the other hand, in the laboratory reference frame the sweeping dominates the 
time evolution in a fixed point of the space and we must use as characteristic 
times of the processes Xn{t) the sweeping times rjf^ = in, i.e., a — 1. 

D.2 Reproducing both the spatial and the temporal properties 

We have now all the ingredients to perform a merging of temporal and spatial 
properties of a turbulent signal in order to define stochastic processes able 
to reproduce in a realistic way both spatial and temporal fluctuations (5.30)- 
(5.31). We just have to merge in a proper way the two previous algorithms. 
For example, for a d-dimensional multi-affine field such as, say, one of the 
three components of a turbulent field in a Lagrangian reference frame we can 
use the following model: 

^4x,t) = E E an,,{t)J^-^). (D.6) 

n=l k=l V -f-n / 

where the temporal dependency of an,k{t) is chosen following the sequential 
algorithm while its spatial part are given by the dyadic structure of the non- 
sequential algorithm. In (D.6) we have used the notation V£,(x, f) in order to 
stress the typical Lagrangian character of such a field. 

We are now able to guess a good candidate for the same field measured in the 
laboratory-reference frame, i.e. where the time properties are dominated by 
the sweeping of small scales by large scales. Indeed, in order to reproduce the 
sweeping effects one needs that the centers {x„ fe} of the wavelets-like functions 
move according a swept-dynamics. 

To do so, let us define the Eulerian model: 

^ 2''^""'' /x-x Jt)\ 

veM = J: E a^At)^ I^^^^] . (D.7) 

n=l k=l \ J 

where the difference with the previous definition is in the temporal depen- 
dency of the centers of the wavelets, x„fc(t). According to the Richardson- 
Kolmogorov cascade picture, one assumes that sweeping is present, i.e., x^^ ^ ~ 
Xn-i,fc' + ^n,k where (n, k') labels the "mother" of the (n, /c)-eddy and r„_fe is 
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a stochastic vector which depends on r„_i and evolves with characteristic 
time T„ oc (inY^^. Furthermore, its norm is 0{£n)'- Ci < |r„_fe|/£n < C2 where 
Ci and C2 are constants of order one. 

We now see that if we measure in one fixed spatial point a fluctuations over 
a time delay At, is like to measure a simultaneous fluctuations at scale sep- 
aration R = UAt, i.e. due to the sweeping the main contribution to the sum 
will be given by the terms with scale-index n = log2(-R = UAt) while the 
temporal dependency of the coefficients {a„.fc(t)} will be practically frozen on 
that time scale. Therefore, one has the proper spatial and temporal statistics, 
see Ref. [4] for details. This happens because in presence of the sweeping the 
main contribution is given by the displacement of the center at large scale, i.e. 
Sro = |ro(t + At) — ro{t)\ ~ UAt, and the eddy turnover time at scale is 
0{{in)^~'^) always large that the sweeping time 0(4t) at the same scale. 
In the previous discussion for the sake of simplicity we did not consider the 
incompressibility condition. However one can take into account this constraint 
by the projection on the solcnoidal space. 

In conclusion we have a way to build up a synthetic signal with the proper 
Eulerian (laboratory) properties, i.e. with sweeping, and also with the proper 
Lagrangian properties. 
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